Summary of On Prompt-driven Safeguarding For Large Language Models, by Chujie Zheng et al.
On Prompt-Driven Safeguarding for Large Language Models
by Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, Nanyun Peng
First submitted to arxiv on: 31 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper explores how large language models (LLMs) respond to queries with safety prompts, aiming to improve their ability to distinguish between harmless and harmful requests. The authors investigate how LLMs process input queries in representation space, finding that safety prompts shift these representations towards a “higher-refusal” direction, making models more likely to refuse assistance even for benign queries. Surprisingly, they also find that LLMs can naturally identify harmless and harmful queries without safety prompts. To optimize safety prompt performance, the authors propose Directed Representation Optimization (DRO), treating safety prompts as trainable embeddings. DRO learns to move query representations along or opposite the refusal direction based on their harmfulness. The proposed method is tested on eight LLMs using out-of-domain and jailbreak benchmarks, demonstrating significant improvements in safeguarding performance without compromising model generalization. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about making large language models safer by changing how they respond to bad requests. Researchers wanted to know why certain words or phrases could make a model say something mean or harmful. They found that adding special “safety prompts” can actually make the model more likely to refuse help, even if the request is okay. But here’s the cool part: the models are actually pretty good at telling when someone is asking for help in a bad way without any special prompts. To make things better, the researchers came up with a new way to adjust these safety prompts so that they work really well and don’t hurt the model’s ability to answer questions correctly. |
Keywords
* Artificial intelligence * Generalization * Optimization * Prompt
