Summary of A Pseudo-semantic Loss For Autoregressive Models with Logical Constraints, by Kareem Ahmed et al.
A Pseudo-Semantic Loss for Autoregressive Models with Logical Constraints
by Kareem Ahmed, Kai-Wei Chang, Guy Van den Broeck
First submitted to arxiv on: 6 Dec 2023
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed approach in this paper bridges the gap between symbolic and neural learning by optimizing the likelihood of a symbolic constraint under an autoregressive output distribution. Traditional methods assume fully-factorized distributions, limiting their applicability to expressive autoregressive models like transformers. To address this, the authors introduce a pseudolikelihood-based approximation centered around a model sample, which is factorized and allows for the reuse of solutions to sub-problems. This approach exhibits low entropy and KL-divergence around the model sample, improving the prediction of logically-consistent outputs in tasks such as Sudoku and shortest-path prediction. Additionally, the authors demonstrate state-of-the-art detoxification results on large language models by steering their outputs away from toxic generations. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper introduces a new approach to combining symbolic and neural learning, which can be used to predict logical outcomes in problems like Sudoku or shortest-path prediction. The method works by optimizing the likelihood of certain rules under an output distribution that is similar to how people think. This helps the model generate more realistic and sensible outputs. The authors tested their approach on some example tasks and found it was very effective. |
Keywords
* Artificial intelligence * Autoregressive * Likelihood
