Summary of Multi-modal Parameter-efficient Fine-tuning Via Graph Neural Network, by Bin Cheng et al.
Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network
by Bin Cheng, Jiaxuan Lu
First submitted to arxiv on: 1 Aug 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a multi-modal parameter-efficient fine-tuning method for foundation models. The approach utilizes graph networks to leverage structural knowledge in downstream tasks, addressing limitations of existing methods that only model single modalities. By processing images and text descriptions through frozen encoders, the model generates image features and text features, which are then used to construct a graph and extract relevant knowledge and relationships. Elastic Weight Consolidation regularization is incorporated to mitigate forgetting during task learning. The proposed method achieves improved test accuracies on the OxfordPets, Flowers102, and Food101 datasets. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper creates a new way for foundation models to learn from multiple sources of information. It’s like a superpower that lets machines understand relationships between images and words! To make this happen, they use special networks called graph networks. These networks take in pictures and text descriptions, then figure out how the features match up. They even add a special trick to help the model remember what it learned before. This new method does better than other methods on some tricky picture-recognition tasks! |
Keywords
* Artificial intelligence * Fine tuning * Multi modal * Parameter efficient * Regularization
