Summary of Evaluation Of Autonomous Systems Under Data Distribution Shifts, by Daniel Sikar et al.
Evaluation of autonomous systems under data distribution shifts
by Daniel Sikar, Artur Garcez
First submitted to arxiv on: 28 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed paper investigates the limitations of using data in autonomous systems as the data distribution shifts. It argues that there is a threshold beyond which control must be relinquished by the system and operation halted or handed to a human operator. The authors demonstrate this concept using a computer vision toy example, showing how network predictive accuracy is impacted by data distribution shifts. They propose distance metrics between training and testing data to define safe operation limits within these shifts. The paper concludes that beyond an empirically obtained threshold of the data distribution shift, it is unreasonable to expect network predictive accuracy not to degrade. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Autonomous systems can only safely use data up to a certain point before things start going wrong. This paper shows that as the data changes, the system’s predictions get worse and worse. The authors use a simple example from computer vision to prove this point. They suggest ways to measure how different the training and testing data are, and when it gets too different, the system should stop making decisions or hand them over to a human. |
