Summary of Otclean: Data Cleaning For Conditional Independence Violations Using Optimal Transport, by Alireza Pirhadi et al.
OTClean: Data Cleaning for Conditional Independence Violations using Optimal Transport
by Alireza Pirhadi, Mohammad Hossein Moslemi, Alexander Cloninger, Mostafa Milani, Babak Salimi
First submitted to arxiv on: 4 Mar 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Databases (cs.DB)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces a framework called that leverages optimal transport theory for data repair under conditional independence (CI) constraints. This is crucial for developing fair and trustworthy machine learning models. The authors formulate the data repair problem as a Quadratically Constrained Linear Program (QCLP) and propose an alternating method to solve it, but this approach faces scalability issues due to the computational cost of computing optimal transport distances like the Wasserstein distance. To address these challenges, they reframe the problem as a regularized optimization problem and develop an iterative algorithm inspired by Sinkhorn’s matrix scaling algorithm. This efficient algorithm can handle high-dimensional and large-scale data. The authors demonstrate the efficacy and efficiency of their proposed methods through extensive experiments, showcasing their practical utility in real-world data cleaning and preprocessing tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper is about making sure machine learning models are fair and trustworthy by controlling how they use data. They created a new way to fix broken data called that uses something called optimal transport theory. This helps keep the important information in the data while removing the bad parts. The problem with their first approach was that it took too long to process large amounts of data, so they came up with a faster solution. This new method is good at handling big datasets and can be used for things like cleaning and preparing data for machine learning models. |
Keywords
* Artificial intelligence * Machine learning * Optimization
