Summary of Mitigating Semantic Leakage in Cross-lingual Embeddings Via Orthogonality Constraint, by Dayeon Ki et al.
Mitigating Semantic Leakage in Cross-lingual Embeddings via Orthogonality Constraint
by Dayeon Ki, Cheonbok Park, Hyunjoong Kim
First submitted to arxiv on: 24 Sep 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel approach to improve parallel data mining in multilingual pre-trained models is proposed by introducing a new training objective called ORthogonAlity Constraint LEarning (ORACLE). The current disentangled representation learning methods suffer from semantic leakage, where language-specific information is unintentionally incorporated into the semantic representations. This limits the effectiveness of retrieving embeddings that accurately represent sentence meaning. To address this challenge, ORACLE combines intra-class clustering and inter-class separation components to enforce orthogonality between semantic and language embeddings. Experimental results on cross-lingual retrieval and semantic textual similarity tasks demonstrate the improved performance of ORACLE in reducing semantic leakage and enhancing semantic alignment within the embedding space. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary A team of researchers developed a new way to make computer models better at understanding sentences from different languages. They found that current methods have a problem where they accidentally include information about language itself, rather than just the meaning of the sentence. This makes it harder for computers to find the right answers when searching for similar sentences. To fix this, the team created a new training method called ORACLE, which helps models keep their semantic (meaning-focused) and linguistic (language-specific) representations separate. They tested ORACLE on two tasks and found that it worked much better than current methods at capturing sentence meaning. |
Keywords
» Artificial intelligence » Alignment » Clustering » Embedding space » Representation learning
