Summary of Ovfoodseg: Elevating Open-vocabulary Food Image Segmentation Via Image-informed Textual Representation, by Xiongwei Wu et al.
OVFoodSeg: Elevating Open-Vocabulary Food Image Segmentation via Image-Informed Textual Representation
by Xiongwei Wu, Sicheng Yu, Ee-Peng Lim, Chong-Wah Ngo
First submitted to arxiv on: 1 Apr 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Multimedia (cs.MM)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary In the field of food computing, accurately segmenting ingredients from images is a challenging task due to variations among similar ingredients, emergence of new ones, and high annotation costs. Existing approaches primarily rely on closed-vocabulary text embeddings, which struggle to handle diverse ingredients. To address these limitations, we introduce OVFoodSeg, a framework that employs an open-vocabulary setting and enhances text embeddings with visual context using vision-language models (VLMs). The framework consists of two modules: FoodLearner, which aligns visual information with textual representations related to food, and the Image-Informed Text Encoder. Our approach is divided into two stages: pre-training of FoodLearner and subsequent learning for segmentation. OVFoodSeg demonstrates a significant improvement, achieving an 4.9% increase in mean Intersection over Union (mIoU) on the FoodSeg103 dataset. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine trying to identify ingredients in food pictures! It’s hard because ingredients can look very similar, new ones are being introduced all the time, and it takes a lot of work to label these images. Most current approaches use a limited set of words and don’t do well with diverse ingredients. To solve this problem, we created OVFoodSeg, a system that uses both visual and textual information. Our approach is divided into two stages: first, we train a model to understand the relationship between pictures and text related to food, then we use this knowledge to segment the ingredients in the images. This new method works much better than previous ones, achieving a significant improvement on a benchmark dataset. |
Keywords
» Artificial intelligence » Encoder
