Summary of T2vsafetybench: Evaluating the Safety Of Text-to-video Generative Models, by Yibo Miao et al.
T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models
by Yibo Miao, Yifan Zhu, Yinpeng Dong, Lijia Yu, Jun Zhu, Xiao-Shan Gao
First submitted to arxiv on: 8 Jul 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Cryptography and Security (cs.CR); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The recent development of Sora has led to a new era in text-to-video (T2V) generation, but this also raises concerns about security risks. The generated videos may contain illegal or unethical content, and there is a lack of comprehensive quantitative understanding of their safety, which poses a challenge to their reliability and practical deployment. To address this gap, we introduce T2VSafetyBench, a new benchmark designed for conducting safety-critical assessments of text-to-video models. We define 12 critical aspects of video generation safety and construct a malicious prompt dataset including real-world prompts, LLM-generated prompts, and jailbreak attack-based prompts. Our evaluation results show that no single model excels in all aspects, with different models showing various strengths. The correlation between GPT-4 assessments and manual reviews is generally high, indicating that there is a trade-off between the usability and safety of text-to-video generative models. This highlights the urgency of prioritizing video safety as the field of video generation rapidly advances. The paper’s findings indicate that as the field of video generation advances, safety risks are set to surge, making it essential to prioritize video safety. We hope that T2VSafetyBench can provide insights for better understanding the safety of video generation in the era of generative AI. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The new Sora technology can create videos from text, but this also raises concerns about what kind of content these videos might include. The researchers wanted to find out how safe these videos are and whether they could be used to spread misinformation or harmful messages. They created a special set of prompts that could trigger unsafe or offensive content and tested different models to see which ones were most likely to produce this type of content. The results show that no one model is perfect, but some do better than others in avoiding unsafe content. The researchers also found that there’s a trade-off between how well the models work and how safe they are. As Sora technology advances, it’s important to prioritize video safety so we can be sure that these videos aren’t used for harm. |
Keywords
* Artificial intelligence * Gpt * Prompt
