Summary of Do No Harm: a Counterfactual Approach to Safe Reinforcement Learning, by Sean Vaskov et al.
Do No Harm: A Counterfactual Approach to Safe Reinforcement Learning
by Sean Vaskov, Wilko Schwarting, Chris L. Baker
First submitted to arxiv on: 19 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Reinforcement Learning (RL) for control has gained popularity due to its ability to learn rich feedback policies. However, when considering safety constraints, constrained optimization approaches penalize agents for constraint violations. This approach is unclear about how much to penalize if agents are initialized in or must visit states where constraint violation might be inevitable. To address this challenge, we formulate a constraint on the counterfactual harm of the learned policy compared to a default, safe policy. Our formulation only penalizes the learner for constraint violations it caused; maintaining feasibility of the optimal control problem. We demonstrate our approach in simulation studies on a rover with uncertain road friction and a tractor-trailer parking environment, showing that agents learn safer policies than contemporary constrained RL methods. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine you’re teaching a machine to make good decisions without crashing or causing harm. This is called Reinforcement Learning (RL). When trying to teach this machine to follow rules, it’s hard to know how much to punish it if it does something bad. To fix this problem, we came up with a new way of thinking about punishment. Instead of just punishing the machine for doing something bad, we consider what would have happened if it had done nothing instead. This helps the machine learn safer decisions faster. We tested our idea in two different scenarios and found that it works better than other methods. |
Keywords
» Artificial intelligence » Optimization » Reinforcement learning
