Summary of Monitoring Latent World States in Language Models with Propositional Probes, by Jiahai Feng et al.
Monitoring Latent World States in Language Models with Propositional Probes
by Jiahai Feng, Stuart Russell, Jacob Steinhardt
First submitted to arxiv on: 27 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A new approach to understanding language models’ internal states is proposed in this paper. The authors hypothesize that language models represent input contexts as latent world models and aim to extract these representations using “propositional probes”. These probes compositionally analyze tokens to bind them into logical propositions, allowing the extraction of world state information. For instance, given the context “Greg is a nurse. Laura is a physicist.”, the model’s activations can be decoded to produce propositions like “WorksAs(Greg, nurse)” and “WorksAs(Laura, physicist)”. The binding subspace plays a crucial role in distinguishing bound tokens (“Greg” and “nurse”) from unbound ones (“Greg” and “physicist”). The authors validate their approach in a closed-world setting with finite predicates and properties. They also demonstrate the generalizability of propositional probes to rewritten short stories and translated contexts. Furthermore, they find that decoded propositions remain faithful in three settings where language models respond unfaithfully: prompt injections, backdoor attacks, and gender bias. This suggests that language models often encode a faithful world model but decode it unfaithfully, motivating the development of better interpretability tools for monitoring LMs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Language models are clever machines that can process human-like text. However, they can sometimes provide incorrect answers or be biased towards certain groups. To improve their performance and fairness, researchers need to understand how these models work internally. This paper proposes a new way to “listen in” on language models’ internal states by analyzing their activations. The authors use special probes that break down words into their constituent parts and bind them together to form logical statements about the world. For example, given the sentence “Greg is a nurse. Laura is a physicist.”, the model’s activations can be decoded to produce statements like “Greg works as a nurse” and “Laura works as a physicist”. This approach is tested in different scenarios and shows promising results. It could help developers create more accurate and unbiased language models. |
Keywords
» Artificial intelligence » Prompt
