Summary of Visual Prompting with Iterative Refinement For Design Critique Generation, by Peitong Duan et al.
Visual Prompting with Iterative Refinement for Design Critique Generation
by Peitong Duan, Chin-Yi Chen, Bjoern Hartmann, Yang Li
First submitted to arxiv on: 22 Dec 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A recent paper proposes an iterative visual prompting approach to automate design critiques in user interface (UI) design. The approach leverages large language models (LLMs) to generate detailed design comments grounded in a given UI image, along with corresponding bounding boxes mapping each comment to specific regions. This is achieved through iterative refinement of text output and visual prompts using few-shot samples tailored for each step. The proposed pipeline was evaluated using LLMs Gemini-1.5-pro and GPT-4o, showing that human experts generally preferred generated design critiques over baselines, with a 50% reduction in the gap from human performance for one rating metric. Additionally, experiments demonstrated the approach’s generalizability to other multimodal tasks, such as open-vocabulary object and attribute detection. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Automating design critiques can significantly improve the efficiency of UI design workflows. A new approach uses large language models (LLMs) to generate detailed design comments that are visually grounded in a given UI image. The method takes an input UI screenshot and design guidelines, then generates a list of design comments along with corresponding bounding boxes. This process is driven by LLMs, which refine both text output and bounding boxes using few-shot samples. The approach was tested with Gemini-1.5-pro and GPT-4o, showing that human experts prefer the generated critiques over baselines. |
Keywords
» Artificial intelligence » Few shot » Gemini » Gpt » Prompting
