Summary of Tarsier: Recipes For Training and Evaluating Large Video Description Models, by Jiawei Wang et al.
Tarsier: Recipes for Training and Evaluating Large Video Description Models
by Jiawei Wang, Liping Yuan, Yuchen Zhang, Haomiao Sun
First submitted to arxiv on: 30 Jun 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Generating high-quality video descriptions is a crucial task in understanding videos. The paper introduces Tarsier, a family of large-scale video-language models that employ CLIP-ViT and LLM to encode frames separately and model temporal relationships. The two-stage training procedure enables the Tarsier models to outperform existing open-source models by 51.4% in human evaluation, comparable to state-of-the-art proprietary models. Additionally, Tarsier proves to be a versatile generalist model, achieving new state-of-the-art results across nine public benchmarks, including multi-choice VQA, open-ended VQA, and zero-shot video captioning. The paper also introduces the DREAM-1K benchmark for evaluating video description models, featuring diverse videos and complexity levels. Tarsier’s capabilities are showcased through its performance on this new benchmark. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine having a computer program that can describe what’s happening in a video with great accuracy! The paper is about creating such a program called Tarsier. It uses special techniques to understand videos and write descriptions of what’s happening in them. Tarsier does a much better job than other programs, and it’s even good at doing other tasks like answering questions and writing captions for videos. The researchers also created a new test to see how well these kinds of programs can do their job. |
Keywords
* Artificial intelligence * Vit * Zero shot
