Summary of Genai-bench: Evaluating and Improving Compositional Text-to-visual Generation, by Baiqi Li et al.
GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation
by Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, Deva Ramanan
First submitted to arxiv on: 19 Jun 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Multimedia (cs.MM)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper presents a study on the performance of leading image and video generation models in responding to compositional text prompts that require attributes, relationships, and higher-order reasoning. The authors conduct a human evaluation study on GenAI-Bench, comparing the results with automated evaluation metrics such as VQAScore, CLIPScore, PickScore, HPSv2, and ImageReward. The findings show that VQAScore outperforms previous metrics in evaluating the generated images and can improve generation by ranking candidate images. The authors also release a new GenAI-Rank benchmark with over 40,000 human ratings to evaluate scoring metrics on ranking images generated from the same prompt. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper investigates how well AI models generate images when given instructions that involve complex ideas like logic and comparison. Researchers tested top image generation models using a special dataset called GenAI-Bench and asked humans to rate the results. They found that one metric, VQAScore, is much better than others at judging if an image matches what it’s supposed to depict. This improvement can be achieved without fine-tuning the models by simply ranking a few top candidate images based on VQAScore. The authors also created a new benchmark with over 80,000 human ratings to help scientists test and compare different AI models and evaluation methods. |
Keywords
» Artificial intelligence » Fine tuning » Image generation » Prompt
