RAG: Retrieval-Augmented Generation enters the literature
In one sentence Lewis et al. at Facebook AI publish the RAG paper, combining a dense retriever (DPR) with a seq2seq generator (BART) to answer knowledge-intensive questions without baking all facts into the weights.
A model like GPT-3 knows many things because it "memorized" them during training, but when asked specific factual questions it often makes mistakes and confabulates. Plus, to update its knowledge you'd have to retrain it.
Facebook proposes a different idea that will become foundational: connect the model to a search engine. When a question arrives, the system first looks up the most relevant documents in a large database, then passes both the question and the retrieved documents to the model. The answer is written based on real information.
For anyone building corporate chatbots or assistants on their own data, this is the reference pattern. It's born here, even though the name "RAG" only enters everyday vocabulary two years later.
Companies
Meta, Facebook AI Research
Tools
RAG, DPR
Tags
Sources