Summary of Gecko: Versatile Text Embeddings Distilled From Large Language Models, by Jinhyuk Lee et al.

Gecko: Versatile Text Embeddings Distilled from Large Language Models

by Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik Duddu, Gustavo Hernandez Abrego, Weiqiang Shi, Nithi Gupta, Aditya Kusupati, Prateek Jain, Siddhartha Reddy Jonnalagadda, Ming-Wei Chang, Iftekhar Naim

First submitted to arxiv on: 29 Mar 2024

GrooveSquid.com Paper Summaries

GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!

Summary difficulty	Written by	Summary
High	Paper authors	High Difficulty Summary Read the original abstract here
Medium	GrooveSquid.com (original content)	Medium Difficulty Summary Gecko is a compact text embedding model that leverages knowledge distillation from large language models (LLMs) to achieve strong retrieval performance. The two-step distillation process involves generating synthetic paired data using an LLM, followed by refining the data quality through retrieving candidate passages and relabeling positive and hard negative passages with the same LLM. Gecko’s compactness is demonstrated on the Massive Text Embedding Benchmark (MTEB), where it outperforms existing entries with 768 embedding size using only 256 embedding dimensions. Additionally, Gecko with 768 embedding dimensions achieves an average score of 66.31, comparable to larger models and higher dimensional embeddings.
Low	GrooveSquid.com (original content)	Low Difficulty Summary Gecko is a new text embedding model that does a great job at finding answers in large amounts of text. It uses ideas from other big language models to help it get better. The process involves creating fake data and then making sure it’s accurate by using the same big language model again. Gecko is special because it can do well even with less information than other models, which makes it useful for computers that need to quickly find answers in a lot of text.