Summary of Ntsebench: Cognitive Reasoning Benchmark For Vision Language Models, by Pranshu Pandya et al.
NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models
by Pranshu Pandya, Vatsal Gupta, Agney S Talwarr, Tushar Kataria, Dan Roth, Vivek Gupta
First submitted to arxiv on: 15 Jul 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Information Retrieval (cs.IR)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed paper introduces NTSEBench, a new dataset designed to evaluate cognitive multi-modal reasoning and problem-solving skills of large language models (LLMs) and vision-language models (VLMs). The dataset consists of 2728 multiple-choice questions accompanied by 4,642 images, categorized into 26 types. These questions are drawn from the nationwide NTSE examination in India and feature a mix of visual and textual general aptitude challenges. State-of-the-art LLMs and VLMs are established as baselines on the dataset, with four distinct modeling strategies proposed to handle different modalities. This paper’s goal is to assess intelligence and critical thinking skills beyond mere rote learning. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary NTSEBench is a new way to test how well big computer models can solve problems that need both language understanding and visual thinking. Right now, these models are great at doing simple tasks that require common sense, but they struggle with more complicated problems that demand deeper understanding. The dataset has 2728 questions, along with 4,642 pictures, and is designed to challenge the models’ critical thinking skills. |
Keywords
» Artificial intelligence » Language understanding » Multi modal
