Summary of Lmm-vqa: Advancing Video Quality Assessment with Large Multimodal Models, by Qihang Ge et al.
LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models
by Qihang Ge, Wei Sun, Yu Zhang, Yunhao Li, Zhongpeng Ji, Fengyu Sun, Shangling Jui, Xiongkuo Min, Guangtao Zhai
First submitted to arxiv on: 26 Aug 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The explosive growth of videos on streaming media platforms highlights the need for effective video quality assessment (VQA) algorithms. However, VQA remains an extremely challenging task due to diverse video content and complex distortions. To address this issue, we propose a Large Multi-Modal Video Quality Assessment (LMM-VQA) model that leverages large multimodal models like GPT-4V for visual understanding tasks. Our approach reformulates the quality regression problem into a question-and-answering task, using Q&A prompts for instruction tuning. We design a spatiotemporal vision encoder to extract spatial and temporal features, which are then mapped into the language space by a projector for modality alignment. The aligned visual tokens and text tokens are aggregated as inputs for the large language model (LLM) to generate quality scores and levels. Our LMM-VQA model achieves state-of-the-art performance on five VQA benchmarks, with an average improvement of 5% in generalization ability over existing methods. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine trying to figure out if a video is good or bad just by looking at it. That’s the challenge that this paper solves. The problem is that videos can be very different and distorted in many ways, making it hard to tell what’s good and what’s not. To solve this issue, the authors use a special kind of model called GPT-4V, which is great at understanding visual information. They reformulate the problem as a question-and-answer task, where the model has to answer questions about video quality based on the content. The result is an algorithm that can accurately assess video quality and even perform well on other tasks like understanding videos in general. This is important because streaming media platforms need better ways to monitor and improve video quality. |
Keywords
» Artificial intelligence » Alignment » Encoder » Generalization » Gpt » Instruction tuning » Large language model » Multi modal » Regression » Spatiotemporal
