Summary of Mobile-bench: An Evaluation Benchmark For Llm-based Mobile Agents, by Shihan Deng et al.
Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents
by Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, Shuo Shang
First submitted to arxiv on: 1 Jul 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary In this paper, researchers tackle the challenge of benchmarking large language model (LLM) based mobile agents in human-computer interaction. The current lack of benchmarks is addressed by proposing Mobile-Bench, a novel evaluation tool that assesses the capabilities of LLM-based mobile agents. To achieve this, the authors expand conventional user interface operations by incorporating APIs to accelerate task completion, collect real-user query data and augment it with language models, and categorize tasks into three groups (SAST, SAMT, MAMT) based on complexity. The Mobile-Bench dataset consists of 832 entries, featuring over 200 tasks designed for evaluating multi-APP collaboration scenarios. Furthermore, the authors introduce a new evaluation metric, CheckPoint, to assess whether mobile agents reach crucial points during their planning and reasoning processes. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper develops a special tool called Mobile-Bench to help evaluate how well large language models work when used in mobile devices. Right now, there aren’t many ways to test this type of technology. The researchers tried to fix this by making the user interface more efficient, collecting real-user data, and grouping tasks into three categories based on difficulty. They also created a dataset with over 200 tasks that test how well these language models work when working together with different apps. This tool can help make better language models in the future. |
Keywords
» Artificial intelligence » Large language model
