Summary of Rebus: a Robust Evaluation Benchmark Of Understanding Symbols, by Andrew Gritsevskiy et al.
REBUS: A Robust Evaluation Benchmark of Understanding Symbols
by Andrew Gritsevskiy, Arjun Panickssery, Aaron Kirtland, Derik Kauffman, Hans Gundlach, Irina Gritsevskaya, Joe Cavanagh, Jonathan Chiang, Lydia La Roux, Michelle Hung
First submitted to arxiv on: 11 Jan 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Computers and Society (cs.CY)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed benchmark evaluates the performance of multimodal large language models on rebus puzzles, requiring image recognition, string manipulation, hypothesis testing, multi-step reasoning, and understanding human cognition. The dataset covers 333 original examples of image-based wordplay, cluing 13 categories such as movies, composers, major cities, and food. GPT-4o significantly outperforms other models, but even the best model has a final accuracy of only 42%, highlighting the need for substantial improvements in reasoning. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Rebus puzzles are a type of wordplay that uses images to represent words or phrases. Researchers created a new benchmark to test how well language models can solve these puzzles. They made a dataset with 333 examples and found that some models were really good at solving them, but not perfect. Even the best model only got 42% correct, which means there’s still room for improvement. |
Keywords
* Artificial intelligence * Gpt
