Summary of An Analysis on Automated Metrics For Evaluating Japanese-english Chat Translation, by Andre Rusli et al.
An Analysis on Automated Metrics for Evaluating Japanese-English Chat Translation
by Andre Rusli, Makoto Shishido
First submitted to arxiv on: 24 Dec 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Medium Difficulty summary: This paper investigates how traditional metrics and neural-based methods evaluate the performance of Neural Machine Translation (NMT) models on chat translation tasks. Specifically, it analyzes BLEU and TER alongside BERTScore and COMET in ranking NMT models for chat translations. The results suggest that all metrics consistently identify the top-performing model. However, when correlating with human-annotated scores, neural-based metrics like COMET outperform traditional methods. Interestingly, even the best metric struggles to score English translations from Japanese sentences with anaphoric zero-pronouns. This study highlights the importance of considering both simplicity and correlation when evaluating NMT models. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Low Difficulty summary: This paper looks at how different ways to measure the quality of chat translation from machines (like Google Translate) work. It compares traditional methods, like BLEU and TER, with newer neural-based methods, such as BERTScore and COMET. The results show that all these methods agree on which machine translator is best. However, when we ask humans to rate the translations, the new neural-based methods do a better job of matching their scores. This study helps us understand how to measure the quality of chat translation from machines. |
Keywords
» Artificial intelligence » Bleu » Translation
