Summary of Chinatravel: a Real-world Benchmark For Language Agents in Chinese Travel Planning, by Jie-jing Shao et al.
ChinaTravel: A Real-World Benchmark for Language Agents in Chinese Travel Planning
by Jie-Jing Shao, Xiao-Wen Yang, Bo-Wen Zhang, Baizhi Chen, Wen-Da Wei, Guohao Cai, Zhenhua Dong, Lan-Zhe Guo, Yu-feng Li
First submitted to arxiv on: 18 Dec 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Recent advances in Large Language Models (LLMs) have led to the development of Language Agents, which are being applied to various domains, including travel planning. The existing benchmarks for this domain lack diversity and do not reflect real-world requirements, making it difficult to deploy these agents effectively. To address this gap, we introduce ChinaTravel, a benchmark specifically designed for authentic Chinese travel planning scenarios. Our benchmark collects travel requirements through questionnaires and proposes a compositional language that enables scalable evaluation. Empirical studies show the potential of neuro-symbolic agents in travel planning, achieving a constraint satisfaction rate of 27.9%, surpassing purely neural models at 2.6%. The findings highlight the significance of ChinaTravel as a milestone for advancing language agents in complex, real-world planning scenarios. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Recently, big computers (LLMs) have helped create helpful tools called Language Agents. One important task these tools do is plan trips. But existing tests for this task are not very good because they don’t reflect how people really ask for help when planning a trip. To fix this problem, we created ChinaTravel, a special test that shows what real people need to plan a trip in China. We collected information from questionnaires and came up with a way to evaluate these Language Agents in a fair way. Our studies show that some computer models can do a good job of planning trips, especially those that combine human thinking with artificial intelligence. This is an important step forward for making language agents more helpful in real-life situations. |
