Summary of Legal-uqa: a Low-resource Urdu-english Dataset For Legal Question Answering, by Faizan Faisal et al.
LEGAL-UQA: A Low-Resource Urdu-English Dataset for Legal Question Answering
by Faizan Faisal, Umair Yousaf
First submitted to arxiv on: 16 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper introduces LEGAL-UQA, a new dataset designed to help develop natural language processing (NLP) resources for low-resource languages like Urdu. Specifically, it focuses on constitutional law, providing 619 question-answer pairs derived from Pakistan’s constitution, along with corresponding legal article contexts in both English and Urdu. The dataset creation process involved OCR extraction, manual refinement, and GPT-4-assisted translation and generation of QA pairs. Experimental results evaluate the latest generalist language and embedding models on LEGAL-UQA, with Claude-3.5-Sonnet achieving 99.19% human-evaluated accuracy. Additionally, the paper fine-tunes mt5-large-UQA-1.0 to highlight challenges in adapting multilingual models to specialized domains. The study also assesses retrieval performance, showing OpenAI’s text-embedding-3-large outperforming Mistral’s mistral-embed. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper creates a special dataset called LEGAL-UQA that helps computers understand Urdu language and law. It has 619 questions and answers about Pakistan’s constitution, along with some context information. The researchers made this dataset by extracting text from books, refining it manually, and using artificial intelligence to translate and create the questions and answers. They tested different computer models on LEGAL-UQA and found that one model, Claude-3.5-Sonnet, was very good at understanding the questions and answering them correctly. The study also shows how computers can be fine-tuned for specific tasks like this one. |
Keywords
» Artificial intelligence » Claude » Embedding » Gpt » Natural language processing » Nlp » Translation
