Summary of Chisafetybench: a Chinese Hierarchical Safety Benchmark For Large Language Models, by Wenjing Zhang et al.
CHiSafetyBench: A Chinese Hierarchical Safety Benchmark for Large Language Models
by Wenjing Zhang, Xuejiao Lei, Zhaoxiang Liu, Meijuan An, Bikun Yang, KaiKai Zhao, Kai Wang, Shiguo Lian
First submitted to arxiv on: 14 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A new benchmark for evaluating the safety of large language models (LLMs) in Chinese contexts is introduced, addressing the scarcity of existing benchmarks and inadequate taxonomies. The CHiSafetyBench dataset covers a hierarchical taxonomy with 5 risk areas and 31 categories, comprising multiple-choice questions and question-answering tasks that evaluate LLMs’ ability to identify risky content and refuse answering risky questions. Automatic evaluation is validated as a substitute for human evaluation, and experiments reveal varying performance across different safety domains, indicating potential for improvement in Chinese safety capabilities. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper creates a special set of tests to check if language models are safe when used with Chinese text. The tests have two parts: multiple-choice questions and answering questions. This helps us understand how well the models can spot harmful content and decide not to answer tricky questions. The results show that different models perform differently, depending on what kind of safety test they’re taking. This means we need to work on making these language models better at staying safe when used with Chinese text. |
Keywords
» Artificial intelligence » Question answering
