Summary of Wordepth: Variational Language Prior For Monocular Depth Estimation, by Ziyao Zeng et al.
WorDepth: Variational Language Prior for Monocular Depth Estimation
by Ziyao Zeng, Daniel Wang, Fengyu Yang, Hyoungseob Park, Yangchao Wu, Stefano Soatto, Byung-Woo Hong, Dong Lao, Alex Wong
First submitted to arxiv on: 4 Apr 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Multimedia (cs.MM)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed paper investigates whether combining two inherently ambiguous modalities, namely 3D reconstruction from a single image and predicting a 3D scene from text description(s), can lead to metric-scaled reconstructions. The researchers focus on monocular depth estimation, where they use a variational framework to learn the distribution of plausible metric reconstructions corresponding to text captions as a prior. To select a specific reconstruction or depth map, they employ a conditional sampler that samples from the latent space of the variational text encoder and decodes it to the output depth map. The approach is trained by alternating between the text and image branches, demonstrating improved performance on indoor (NYUv2) and outdoor (KITTI) scenarios. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper explores how combining two ambiguous tasks can lead to better results. It tries to predict 3D scenes from single images or text descriptions, but finds that this is tricky because scale is hard to determine. Instead, it looks at how language and vision can work together to get a more accurate picture. By using a special kind of math called variational frameworks, the researchers learn what different text descriptions could mean in terms of 3D scenes. Then, they use this information to choose the best depth map from an image. The results show that using language can make predictions better. |
Keywords
* Artificial intelligence * Depth estimation * Encoder * Latent space
