Summary of Android in the Zoo: Chain-of-action-thought For Gui Agents, by Jiwen Zhang et al.
Android in the Zoo: Chain-of-Action-Thought for GUI Agents
by Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, Duyu Tang
First submitted to arxiv on: 5 Mar 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Computer Vision and Pattern Recognition (cs.CV); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents Chain-of-Action-Thought (CoAT), a novel approach for predicting sequences of actions in autonomous GUI agents for smartphones. CoAT takes into account the description of previous actions, the current screen, and action thinking to improve action prediction compared to existing methods. The authors demonstrate the effectiveness of CoAT using three off-the-shelf large language models (LLMs) in a zero-shot setting. They also introduce a new dataset, Android-In-The-Zoo (AitZ), which contains 18,643 screen-action pairs with chain-of-action-thought annotations. Fine-tuning a 1B model on this dataset achieves comparable performance to CogAgent-Chat-18B. This work contributes to the development of more advanced autonomous GUI agents for smartphones. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine you’re using your smartphone, and it can complete tasks without you touching the screen. The paper introduces a new way to help your phone do this by understanding what actions it should take next. They call this approach Chain-of-Action-Thought. To make it work, they consider what happened before, what’s on the screen now, and what might happen if they choose one action over another. They tested their method using three different language models and a dataset of 18,643 screen-action pairs. The results show that their method is just as good as some others that were trained for longer. |
Keywords
* Artificial intelligence * Fine tuning * Zero shot
