Summary of Fine-grained and Multi-dimensional Metrics For Document-level Machine Translation, by Yirong Sun et al.
Fine-Grained and Multi-Dimensional Metrics for Document-Level Machine Translation
by Yirong Sun, Dawei Zhu, Yanjun Chen, Erjia Xiao, Xinghao Chen, Xiaoyu Shen
First submitted to arxiv on: 28 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates the capability of instruction-tuned large language models (LLMs) for document-level machine translation (docMT). Unlike prior approaches, this method directly prompts LLMs to translate entire documents in a single pass. The results show that this approach improves translation quality compared to translating sentences separately, even without document-level fine-tuning. However, the advantage is not reflected in BLEU scores, which often favor sentence-based translations. To address this, the paper proposes using the LLM-as-a-judge paradigm for evaluation, where GPT-4 assesses document coherence, accuracy, and fluency. The study demonstrates that instruction-tuned LLMs can effectively leverage document context for translation, but cautions against using BLEU scores to evaluate docMT due to misleading outcomes. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how well large language models (LLMs) can translate long documents all at once. This is different from most studies which focus on translating single sentences. The researchers found that LLMs can do a better job of translation when they’re given the whole document to work with, even if they haven’t been trained specifically for this task. However, some popular ways of measuring how good the translations are don’t always show this improvement. To fix this, the researchers came up with a new way to evaluate the translations using another LLM as a judge. This helps to get a better sense of whether the translations make sense and are accurate. Overall, the study shows that LLMs can be very good at translating documents if they’re given the right instructions. |
Keywords
» Artificial intelligence » Bleu » Fine tuning » Gpt » Translation
