Summary of Adaptagent: Adapting Multimodal Web Agents with Few-shot Learning From Human Demonstrations, by Gaurav Verma et al.
AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations
by Gaurav Verma, Rachneet Kaur, Nishan Srishankar, Zhen Zeng, Tucker Balch, Manuela Veloso
First submitted to arxiv on: 20 Nov 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computation and Language (cs.CL); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed AdaptAgent framework enables multimodal web agents to adapt to new websites and domains using few human demonstrations (up to 2), boosting task success rate by 3.36% to 7.21% over non-adapted state-of-the-art models, corresponding to a relative increase of 21.03% to 65.75%. The framework uses in-context demonstrations for proprietary models and meta-adaptation demonstrations for meta-learned open-weights models, leveraging Multimodal Large Language Models (MLLMs) as the underlying architecture. The authors demonstrate the effectiveness of multimodal demonstrations over text-only ones and analyze the influence of different data selection strategies during meta-learning on the generalization of the agent. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Multimodal web agents are becoming increasingly powerful, but they still struggle to automate tasks on unseen websites and domains. Researchers propose a new approach called AdaptAgent that helps these agents adapt to new situations using just a few examples. This is achieved by fine-tuning the agents’ language models using human demonstrations. The authors tested their idea on two popular benchmarks and found that it improved task success rates by 21-65% compared to previous methods. |
Keywords
* Artificial intelligence * Boosting * Fine tuning * Generalization * Meta learning
