Retrieval-augmented generation

'''Retrieval-augmented generation''' ('''RAG''') is a technique that enables large language models (LLMs) to retrieve and incorporate new information from external data sources.<ref name="IBM-2023">{{Cite web |date=22 August 2023 |title=What is retrieval-augmented generation? |url=https://research.ibm.com/blog/retrieval-augmented-generation-RAG |access-date=7 March 2025 |website=IBM}}</ref> With RAG, LLMs first refer to a specified set of documents, then respond to user queries. These documents supplement information from the LLM's pre-existing training data.<ref name="MIT Technology Review-2024" /> This allows LLMs to use domain-specific and/or updated information that is not available in the training data.<ref name="MIT Technology Review-2024" /> For example, this enables LLM-based chatbots to access internal company data or generate responses based on authoritative sources.

RAG improves LLMs by incorporating information retrieval before generating responses.<ref name="BUZBP" /> Unlike LLMs that rely on static training data, RAG pulls relevant text from databases, uploaded documents, or web sources.<ref name="IBM-2023" /> According to ''Ars Technica'', "RAG is a way of improving LLM performance, in essence by blending the LLM process with a web search or other document look-up process to help LLMs stick to the facts." This method helps reduce AI hallucinations,<ref name="BUZBP" /> which have caused chatbots to describe policies that don't exist, or recommend nonexistent legal cases to lawyers that are looking for citations to support their arguments.<ref name="Ars Technica-2024">{{Cite web |date=6 June 2024 |title=Can a technology called RAG keep AI models from making stuff up? |url=https://arstechnica.com/ai/2024/06/can-a-technology-called-rag-keep-ai-models-from-making-stuff-up/ |access-date=7 March 2025 |website=Ars Technica}}</ref>

RAG also reduces the need to retrain LLMs with new data, saving on computational and financial costs.<ref name="IBM-2023" /> Beyond efficiency gains, RAG also allows LLMs to include sources in their responses, so users can verify the cited sources. This provides greater transparency, as users can cross-check retrieved content to ensure accuracy and relevance.

The term retrieval-augmented generation (RAG) was introduced in a 2020 paper that described combining a parametric language model with a non-parametric external memory accessed through retrieval at inference time.<ref name="BUZBP" />

== RAG and LLM limitations == LLMs can provide incorrect information. For example, when Google first demonstrated its LLM tool "Google Bard" (later re-branded to Gemini), the LLM provided incorrect information about the James Webb Space Telescope. This error contributed to a $100 billion decline in Google's stock value.<ref name="Ars Technica-2024" /> RAG is used to prevent these errors, but it does not solve all the problems. For example, LLMs can generate misinformation even when pulling from factually correct sources if they misinterpret the context. ''MIT Technology Review'' gives the example of an AI-generated response stating, "The United States has had one Muslim president, Barack Hussein Obama." The model retrieved this from an academic book rhetorically titled ''Barack Hussein Obama: America's First Muslim President?'' The LLM did not "know" or "understand" the context of the title, generating a false statement.<ref name="MIT Technology Review-2024" />

LLMs with RAG are programmed to prioritize new information. This technique has been called "prompt stuffing." Without prompt stuffing, the LLM's input is generated by a user; with prompt stuffing, additional relevant context is added to this input to guide the model's response. This approach provides the LLM with key information early in the prompt, encouraging it to prioritize the supplied data over pre-existing training knowledge.<ref>{{Cite web |date=20 June 2024 |title=Mitigating LLM hallucinations in text summarisation |url=https://www.bbc.co.uk/rd/articles/2024-06-mitigating-llm-hallucinations-in-text-summarisation |access-date=7 March 2025 |website=BBC}}</ref>

==Process== Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating an information-retrieval mechanism that allows models to access and utilize additional data beyond their original training set. ''Ars Technica'' notes that "when new information becomes available, rather than having to retrain the model, all that's needed is to augment the model's external knowledge base with the updated information" ("augmentation").<ref name="Ars Technica-2024" /> IBM states that "in the generative phase, the LLM draws from the augmented prompt and its internal representation of its training data to synthesize" an answer.<ref name="IBM-2023" />

=== RAG key stages ===

thumb|Overview of RAG process, combining external documents and user input into an LLM prompt to get tailored output

Typically, the data to be referenced is converted into LLM embeddings, numerical representations in the form of a large vector space. RAG can be used on unstructured (usually text), semi-structured, or structured data (for example knowledge graphs). These embeddings are then stored in a vector database to allow for document retrieval.

Given a user query, a document retriever is first called to select the most relevant documents that will be used to augment the query.<ref name="MIT Technology Review-2024">{{Cite web |date=31 May 2024 |title=Why Google's AI Overviews gets things wrong |url=https://www.technologyreview.com/2024/05/31/1093019/why-are-googles-ai-overviews-results-so-bad/ |access-date=7 March 2025 |website=MIT Technology Review}}</ref><ref name="BUZBP" /> This comparison can be done using a variety of methods, which depend in part on the type of indexing used.<ref name="IBM-2023" />

The model feeds this relevant retrieved information into the LLM via prompt engineering of the user's original query. Newer implementations ({{as of|2023|lc=y}}) can also incorporate specific augmentation modules with abilities such as expanding queries into multiple domains and using memory and self-improvement to learn from previous retrievals.

Finally, the LLM can generate output based on both the query and the retrieved documents.<ref name="MIT Technology Review-2024" /><ref name="BUZBP"/> Some models incorporate extra steps to improve output, such as the re-ranking of retrieved information, context selection, and fine-tuning.

== Applications == Retrieval-augmented generation is used in applications where generated responses need to be grounded in external or frequently updated information. Commonly cited use cases include search engines, question-answering systems, customer support chatbots, enterprise knowledge assistants, content generation, recommendation systems, retail and e-commerce, and industrial or manufacturing workflows.<ref>{{Cite web |title=Use cases for Retrieval-Augmented Generation |url=https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-use-cases.html |website=Amazon Web Services |access-date=2026-05-05}}</ref><ref>{{Cite web |title=What is retrieval-augmented generation? |url=https://www.ibm.com/think/topics/retrieval-augmented-generation |website=IBM |access-date=2026-05-05}}</ref>

In healthcare, RAG has been studied as a way to ground large language model outputs in external medical knowledge sources, although reviews have noted continuing challenges around evaluation, ethics, and clinical reliability.<ref>{{Cite journal |last1=Amugongo |first1=Lameck Mbangula |last2=Mascheroni |first2=Pietro |last3=Brooks |first3=Steven |last4=Doering |first4=Stefan |last5=Seidel |first5=Jan |title=Retrieval augmented generation for large language models in healthcare: A systematic review |journal=PLOS Digital Health |date=2025-06-11 |volume=4 |issue=6 |article-number=e0000877 |doi=10.1371/journal.pdig.0000877 |doi-access=free|pmc=12157099 }}</ref>

== Improvements == Improvements to the basic process above can be applied at different stages in the RAG flow.

=== Encoder === These methods focus on the encoding of text as either dense or sparse vectors. Sparse vectors, which encode the identity of a word, are typically dictionary-length and contain mostly zeros. Dense vectors, which encode meaning, are more compact and contain fewer zeros. Various enhancements can improve the way similarities are calculated in the vector stores (databases).<ref name="Luan-2021">{{Cite journal |last1=Luan |first1=Yi |last2=Eisenstein |first2=Jacob |last3=Toutanova |first3=Kristina |last4=Collins |first4=Michael |date=26 April 2021 |title=Sparse, Dense, and Attentional Representations for Text Retrieval |url=https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00369/100684/Sparse-Dense-and-Attentional-Representations-for |access-date=15 March 2025 |journal=Transactions of the Association for Computational Linguistics|volume=9 |pages=329–345 |doi=10.1162/tacl_a_00369 |arxiv=2005.00181 }}</ref>

* Performance improves by optimizing how vector similarities are calculated. Dot products enhance similarity scoring, while approximate nearest neighbor (ANN) searches improve retrieval efficiency over K-nearest neighbors (KNN) searches.<ref>{{Cite web |date=10 January 2025 |title=Information retrieval |url=https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-information-retrieval |access-date=15 March 2025 |website=Microsoft}}</ref> * Accuracy may be improved with Late Interactions, which allow the system to compare words more precisely after retrieval. This helps refine document ranking and improve search relevance.<ref name="colbert" /> * Hybrid vector approaches may be used to combine dense vector representations with sparse one-hot vectors, taking advantage of the computational efficiency of sparse dot products over dense vector operations.<ref name="Luan-2021" /> * Other retrieval techniques focus on improving accuracy by refining how documents are selected. Some retrieval methods combine sparse representations, such as SPLADE, with query expansion strategies to improve search accuracy and recall.<ref>{{Cite web |last1=Wang |first1=Yup |last2=Conroy |first2=John M. |last3=Molino |first3=Neil |last4=Yang |first4=Julia |last5=Green |first5=Mike |date=2024 |title=Laboratory for Analytic Sciences in TREC 2024 Retrieval Augmented Generation Track |url=https://trec.nist.gov/pubs/trec33/index.html |access-date=15 March 2025 |website=NIST TREC 2024}}</ref>

=== Retriever-centric methods === These methods aim to enhance the quality of document retrieval in vector databases:

* Pre-training the retriever using the ''Inverse Cloze Task'' (ICT), a technique that helps the model learn retrieval patterns by predicting masked text within documents.<ref name="orqa" /> *Supervised retriever optimization aligns retrieval probabilities with the generator model's likelihood distribution. This involves retrieving the top-k vectors for a given prompt, scoring the generated response's perplexity, and minimizing KL divergence between the retriever's selections and the model's likelihoods to refine retrieval.<ref>{{Cite book |date=June 2024 |chapter-url=https://aclanthology.org/2024.naacl-long.463/ |access-date=16 March 2025 |pages=8371–8384 |doi=10.18653/v1/2024.naacl-long.463 |arxiv=2301.12652 |last1=Shi |first1=Weijia |last2=Min |first2=Sewon |last3=Yasunaga |first3=Michihiro |last4=Seo |first4=Minjoon |last5=James |first5=Rich |last6=Lewis |first6=Mike |last7=Zettlemoyer |first7=Luke |last8=Yih |first8=Wen-tau |title=Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) |chapter=REPLUG: Retrieval-Augmented Black-Box Language Models }}</ref> * Reranking techniques can refine retriever performance by prioritizing the most relevant retrieved documents during training.<ref>{{Cite journal |date=2023 |url=https://aclanthology.org/2023.tacl-1.75/ |access-date=16 March 2025 |doi=10.1162/tacl_a_00605 |arxiv=2302.00083 |last1=Ram |first1=Ori |last2=Levine |first2=Yoav |last3=Dalmedigos |first3=Itay |last4=Muhlgay |first4=Dor |last5=Shashua |first5=Amnon |last6=Leyton-Brown |first6=Kevin |last7=Shoham |first7=Yoav |title=In-Context Retrieval-Augmented Language Models |journal=Transactions of the Association for Computational Linguistics |volume=11 |pages=1316–1331 }}</ref> === Language model ===

{{Image frame | width=300 | align=center | content=300px | caption=Retro language model for RAG. Each Retro block consists of Attention, Chunked Cross Attention, and Feed Forward layers. Black-lettered boxes show data being changed, and blue lettering shows the algorithm performing the changes. }}

By redesigning the language model with the retriever in mind, a 25-time smaller network can get comparable perplexity as its much larger counterparts.<ref name="borgeaud"/> Because it is trained from scratch, this method (Retro) incurs the high cost of training runs that the original RAG scheme avoided. The hypothesis is that by giving domain knowledge during training, Retro needs less focus on the domain and can devote its smaller weight resources only to language semantics. The redesigned language model is shown here.

It has been reported that Retro is not reproducible, so modifications were made to make it so. The more reproducible version is called Retro++ and includes in-context RAG.<ref name="wang2023a" />

=== Chunking === {{See also|Chunking (computing)}} Chunking involves various strategies for breaking up the data into vectors so the retriever can find details in it.

{{Image frame | align=center | width=500 | content=500px | caption= Different data styles have patterns that correct chunking can take advantage of. }} Three types of chunking strategies are:{{Citation Needed|date=August 2025}}

* Fixed length with overlap. This is fast and easy. Overlapping consecutive chunks helps to maintain semantic context across chunks. * Syntax-based chunks can break the document up into sentences. Libraries such as spaCy or NLTK can also help. * File format-based chunking. Certain file types have natural chunks built in, and it's best to respect them. For example, code files are best chunked and vectorized as whole functions or classes. HTML files should leave <nowiki><table> or base64 encoded <img></nowiki> elements intact. Similar considerations should be taken for pdf files. Libraries such as Unstructured or LangChain can assist with this method.

=== Hybrid search === Sometimes vector database searches can miss key facts needed to answer a user's question. One way to mitigate this is to do a traditional text search, add those results to the text chunks linked to the retrieved vectors from the vector search, and feed the combined hybrid text into the language model for generation.<ref>{{cite journal|last1=Bruch|first1=Sebastian|last2=Gai|first2=Siyu|last3=Ingber|first3=Amir|title=An Analysis of Fusion Functions for Hybrid Retrieval|journal=ACM Transactions on Information Systems|volume=42|issue=1|year=2023|pages=1-35|doi=10.1145/3596512|arxiv=2210.11934}}</ref>

The adoption of RAG in consumer-facing web search products has given rise to new content optimization disciplines, as practitioners have noted that content retrievability in RAG systems depends on factors like semantic structure, passage-level authority signals, and entity clarity rather than traditional search ranking signals such as backlinks.<ref>{{cite web |last=Yesilyurt |first=Metehan |title=SEO vs. AI search: 101 questions that keep me up at night |url=https://searchengineland.com/seo-ai-search-questions-464550 |website=Search Engine Land |date=November 11, 2025 |access-date=March 24, 2026}}</ref><ref>{{cite web |last=Aggarwal |first=Priyanshu |display-authors=etal |title=GEO: Generative Engine Optimization |url=https://arxiv.org/abs/2311.09735 |website=ArXiv |date=2023}}</ref>

=== Evaluation and benchmarks === RAG systems are commonly evaluated using benchmarks designed to test retrievability, retrieval accuracy and generative quality. Popular evaluation resources include BEIR, a heterogeneous benchmark for zero-shot information retrieval across multiple datasets, and Natural Questions, a large-scale question answering dataset released by Google Research.<ref>{{cite web |last1=Thakur |first1=Nandan |last2=Reimers |first2=Nils |last3=Rücklé |first3=Andreas |last4=Srivastava |first4=Abhishek |last5=Gurevych |first5=Iryna |title=BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models |url=https://arxiv.org/abs/2104.08663 |website=arXiv |date=2021-04-19 |access-date=2026-05-05}}</ref><ref>{{cite web |last1=Kwiatkowski |first1=Tom |last2=Palomaki |first2=Jennimaria |last3=Redfield |first3=Olivia |last4=Collins |first4=Michael |last5=Parikh |first5=Ankur |last6=Alberti |first6=Chris |last7=Epstein |first7=Danielle |last8=Polosukhin |first8=Illia |last9=Devlin |first9=Jacob |last10=Lee |first10=Kenton |last11=Toutanova |first11=Kristina |last12=Jones |first12=Llion |last13=Kelcey |first13=Matthew |last14=Chang |first14=Ming-Wei |last15=Dai |first15=Andrew M. |last16=Uszkoreit |first16=Jakob |last17=Le |first17=Quoc |last18=Petrov |first18=Slav |title=Natural Questions: a Benchmark for Question Answering Research |url=https://research.google/pubs/natural-questions-a-benchmark-for-question-answering-research/ |website=Google Research |access-date=2026-05-05}}</ref>

==Challenges== RAG does not prevent hallucinations in LLMs. According to ''Ars Technica'', "It is not a direct solution because the LLM can still hallucinate around the source material in its response."<ref name="Ars Technica-2024" />

While RAG improves the accuracy of large language models (LLMs), it does not eliminate all challenges. One limitation is that while RAG reduces the need for frequent model retraining, it does not remove it entirely. Additionally, LLMs may struggle to recognize when they lack sufficient information to provide a reliable response. Without specific training, models may generate answers even when they should indicate uncertainty. According to IBM, this issue can arise when the model lacks the ability to assess its own knowledge limitations.<ref name="IBM-2023" />

=== RAG poisoning === RAG systems may retrieve factually correct but misleading sources, leading to errors in interpretation. In some cases, an LLM may extract statements from a source without considering its context, resulting in an incorrect conclusion. Additionally, when faced with conflicting information, RAG models may struggle to determine which source is accurate. The worst case outcome of this limitation is that the model may combine details from multiple sources producing responses that merge outdated and updated information in a misleading manner. According to the ''MIT Technology Review'', these issues occur because RAG systems may misinterpret the data they retrieve.<ref name="MIT Technology Review-2024" />

==References== <references>

<ref name="BUZBP">{{Cite journal |last1=Lewis |first1=Patrick |last2=Perez |first2=Ethan |last3=Piktus |first3=Aleksandra |last4=Petroni |first4=Fabio |last5=Karpukhin |first5=Vladimir |last6=Goyal |first6=Naman |last7=Küttler |first7=Heinrich |last8=Lewis |first8=Mike |last9=Yih |first9=Wen-tau |last10=Rocktäschel |first10=Tim |last11=Riedel |first11=Sebastian |last12=Kiela |first12=Douwe |date=2020 |title=Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks |url=https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=33 |pages=9459–9474 |arxiv=2005.11401}}</ref> <ref name="orqa">{{Cite web | url=https://aclanthology.org/P19-1612.pdf | title="Latent Retrieval for Weakly Supervised Open Domain Question Answering" | last1=Lee | first1=Kenton | last2=Chang | first2=Ming-Wei | last3=Toutanova | first3=Kristina | date=2019}}</ref> <ref name="colbert">{{Cite book |last1=Khattab |first1=Omar |last2=Zaharia |first2=Matei |date=2020 |title= Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval|chapter=ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT |chapter-url=https://dl.acm.org/doi/10.1145/3397271.3401075 |pages=39–48 |doi=10.1145/3397271.3401075 |isbn=978-1-4503-8016-4 |chapter-url-access=subscription }}</ref> <ref name="borgeaud">{{cite web | url=https://proceedings.mlr.press/v162/borgeaud22a/borgeaud22a.pdf | title=Improving language models by retrieving from trillions of tokens | last1=Borgeaud | first1=Sebastian | last2=Mensch | first2=Arthur | date=2021}}</ref> <ref name="wang2023a">{{Cite book |last1=Wang |first1=Boxin |last2=Ping |first2=Wei |last3=Xu |first3=Peng |last4=McAfee |first4=Lawrence |last5=Liu |first5=Zihan |last6=Shoeybi |first6=Mohammad |last7=Dong |first7=Yi |last8=Kuchaiev |first8=Oleksii |last9=Li |first9=Bo |last10=Xiao |first10=Chaowei |last11=Anandkumar |first11=Anima |last12=Catanzaro |first12=Bryan |title=Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing |date=2023 |chapter=Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study |chapter-url=https://aclanthology.org/2023.emnlp-main.482/ |pages=7763–7786 |doi=10.18653/v1/2023.emnlp-main.482 |doi-access=free }}</ref> </references>

Category:Large language models Category:Natural language processing Category:Information retrieval systems Category:Generative AI