RAG: Giving LLMs an External Brain

August 12, 2025 · #llm

In the period before 2020, developing with large language models like BERT and the early GPT series was defined by their impressive capabilities and their critical flaws. On one hand, they could process and generate text with surprising quality. On the other hand, they had two fundamental limitations that made them unreliable for many real-world applications.

Functionally, these models were like powerful CPUs with a fixed, read-only memory. Their two main problems were:

1. Static Knowledge, or knowledge cutoff wall: The model’s information was frozen at the time of its training. It was completely unaware of any event, data, or discovery that occurred after that date.

2. Hallucination: The models would often generate plausible but incorrect information, a phenomenon we call “hallucination.” Since they delivered these falsehoods with high confidence, they could not be trusted for tasks requiring factual accuracy.

The solution to these problems came from bridging two distinct fields of computer science: Information Retrieval (IR) and Natural Language Processing (NLP). For decades, IR’s primary function was locating relevant documents from a large corpus using keyword-based algorithms like TF-IDF and BM25. In parallel, the NLP field was concerned with processing the language itself.

The innovation was to architect a system that connected them. A 2020 paper from Facebook AI Research, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” formalized this architecture and introduced the term RAG. The core idea was logical and powerful: instead of trying to force a model to memorize all facts internally, connect it to an external, up-to-date knowledge base and give it a mechanism to query that information before generating an answer.

This new architecture changes the information flow. The standard, unreliable process was:

Query → LLM → Answer (Potentially Incorrect)

RAG introduces a new, more robust system:

By making this architectural change, we shift the core task. The model is no longer being asked to recall a fact from memory. Instead, it is given a reading comprehension task: answer a question based on a set of source documents provided in real-time. This simple shift dramatically improves factual accuracy and, just as importantly, provides a clear path to citing sources. In the following sections, we will analyze the components of this system, from indexing the data to the final generation step.

Part I: The Indexing Pipeline - Building the Library of Truth

Before a user can ask a single question, we have to perform the digital equivalent of building and organizing a grand library. This is the offline, foundational work of a RAG system. It’s where we take a chaotic mass of information—PDFs, websites, company docs—and transform it into a highly structured, machine-searchable knowledge base. Get this part wrong, and the entire system will fail. Get it right, and you’ve built an external brain.

The process boils down to three core steps: chopping up the documents (chunking), turning the pieces into math (embeddings), and creating a super-fast card catalog (Vector database).

The Art of the Chunk: Finding the “Goldilocks” Passage

When we first started, our instinct was to do the simplest thing: take a whole document, treat it as one piece, and create a single vector for it. It was a spectacular failure. This approach ran into two huge problems. First, most documents were too large for the model’s limited “attention span” (its context window). Second, a long document discussing ten different topics creates a “vector soup”—an embedding so generic it doesn’t represent any single idea well. The signal was lost in the noise.

We quickly learned that the unit of retrieval had to be smaller than a document. Thus began the art of chunking.

• The Sledgehammer Approach: Fixed-Size Chunks. Our next attempt was a brute-force chop. We wrote simple scripts to split a document every 1,000 characters. It was an improvement, but a clumsy one. We were constantly cutting sentences in half, separating a question from its answer, and generally mangling the logical flow of the text. It was fast, but dumb.

• The AI-Powered approach: Semantic Chunking. Today, the state-of-the-art feels like a bit of magic. Why guess where a topic ends when you can ask an AI? Semantic chunking uses an embedding model to measure the conceptual distance between sentences. When it detects a “semantic break”—a point where the topic shifts—it creates a new chunk. It’s like using a tiny AI to prepare the data for the big AI.

Embeddings: Giving Geometry to Meaning

With our perfectly sized chunks in hand, we faced the next challenge: how do you “search” text? You can’t. You can only search numbers. We needed to translate the semantic meaning of our text chunks into a mathematical format.

The world of NLP had been chasing this for years. The first time it truly felt like science fiction was in 2013 with Google’s Word2Vec. Then, the real unlock for RAG came in 2018 with the arrival of Transformer models like BERT. Unlike its predecessors, BERT created contextual embeddings. The vector for “bank” would be completely different depending on the sentence it was in. This was it. This was what we needed to capture the specific meaning of our text chunks with high fidelity.

The search itself is then surprisingly simple geometry. We take the user’s query, create a vector for it using the same model, and then find the chunk vectors that are “pointing in the same direction” in this high-dimensional space. The standard tool for this is Cosine Similarity.

Vector Databases: The Librarian on Roller Skates

So, we have millions of beautiful vectors. A user query comes in, we embed it, and… now we have an “Oh, no” moment. To find the best match, do we have to run a for loop comparing our query vector to every single one of the millions of chunk vectors in our database? For any real-time application, that’s a non-starter. The latency would be abysmal.

This is where we cheat. Brilliantly.

We don’t need the perfect nearest neighbor. We just need the pretty-darn-good nearest neighbors, and we need them fast. This is the job of Approximate Nearest Neighbor (ANN) search algorithms. And the undisputed champion in this space is an algorithm called HNSW (Hierarchical Navigable Small World).

Think of HNSW as a GPS for your vector space. Instead of checking every house (vector) on the map, it builds a multi-layered network. The search starts on the top layer, a super-sparse “interstate highway” system that lets it jump to the right continent-sized region of the vector space in a single hop. Then it drops down to a “state highway” layer to find the right city, then to the “local roads” to find the right neighborhood, until it finally pinpoints the exact vectors it needs on the most detailed bottom layer.

This clever, hierarchical navigation is the engine inside modern vector databases like Pinecone, Weaviate, or Milvus, and high-performance libraries like Meta’s FAISS. They trade an infinitesimal amount of accuracy for a colossal gain in speed, reducing search times from minutes to milliseconds.

Part II: The Retrieval Pipeline — The Moment of Truth

All the heavy lifting in Part I (the chunking, embedding, and indexing) was just the setup. It’s the behind-the-scenes work of building the theater. Now, the show is about to start. The user types a question and hits “Enter.” The clock starts ticking.

Everything that happens next must occur in milliseconds. This is the online, real-time pipeline where we retrieve relevant knowledge and generate a factually grounded answer. Let’s walk through the journey of a single query, using one of your article topics as an example: “How does continuous batching improve LLM throughput?”

Step 1: The Query Becomes a Vector

First, the user’s question isn’t treated as simple text. It’s fed into the exact same embedding model we used to index our documents in Part I. This is a non-negotiable, critical step. You cannot use one model for your library and a different one for the query. Why? Because the models must share the same “semantic coordinate system.” The query vector and the document vectors must exist in the same high-dimensional space for any comparison to be meaningful.

So, the string "How does continuous batching improve LLM throughput?" is transformed into a vector, let’s call it v_query.

Step 2: The Search — Ask the Librarian

Now, v_query is sent to our vector database—the hyper-efficient librarian we built. The database takes the query vector and its HNSW index springs to life. It navigates the layers of the graph and, in a flash, returns a ranked list of the “nearest neighbors.”

It doesn’t just return the single best match. We configure it to return the top-k most similar chunks—let’s say, k=5. What we get back isn’t the vectors themselves, but pointers to the five original text chunks from our documents whose embeddings were closest to our query vector. These are our “source materials” for the answer.

Step 3: Augmentation — The “Open-Book Exam”

This is the magic trick, the “A” in RAG. We don’t just pass the user’s question to the LLM. That would put us right back where we started, relying on the model’s internal, potentially outdated memory.

Instead, we perform prompt augmentation. We construct a brand-new, detailed prompt that gives the LLM everything it needs to answer the question, and nothing it doesn’t. We “stuff” the raw text of the top 5 retrieved chunks directly into the prompt’s context window, along with a very clear set of instructions.

Step 4: Generation — “What the LLM Actually Sees”

The final prompt we send to the LLM looks nothing like the user’s original, simple question. It’s a carefully engineered command that looks something like this:

SYSTEM: You are a helpful assistant. You must answer the user's question based *only* on the context provided below. Do not use any external knowledge. If the information to answer the question is not in the context, you must state that you cannot answer.

CONTEXT:
---
[Chunk 34: (Text from your article) "...Continuous batching works by grouping new requests into a batch on the fly. Unlike static batching, it doesn't wait for the entire batch to finish before starting new work, which dramatically reduces GPU idle time..."]
---
[Chunk 12: (Text from another source) "...the primary benefit is increased throughput, measured in tokens per second. By ensuring the GPU is always processing, continuous batching can lead to a 20x or higher improvement in overall throughput..."]
---
[Chunk 5: (Text from a system design doc) "...key to this process is a scheduler that can manage incoming requests of varying lengths and add them to the running batch as soon as processing capacity becomes available..."]
---
(and so on for the other retrieved chunks)

USER: How does continuous batching improve LLM throughput?

This changes the game entirely. We have transformed the task. We are no longer asking the LLM, “What do you remember about continuous batching?” We are asking it, “Here is a document. Read it and answer this question based on its contents.”

Part III: The Frontier, Fine-Tuning the External Brain

We’ve now built a complete, functional RAG system. It’s a huge step up from a standalone LLM. But as any engineer knows, v1.0 is never the end of the story. Once a system is running, you immediately start finding its edges, its limitations, and all the ways you can make it better. The frontier of RAG research is focused on exactly that: making the retrieval process smarter, more precise, and more resilient.

Here are three key advancements that are moving RAG systems from simply “good” to truly production-grade.

A. Hybrid Search: The Best of Both Worlds

Our shiny new vector search system is fantastic at understanding semantic meaning. A user can ask, “How do I make my car faster?” and it will correctly retrieve documents about “engine tuning” and “performance upgrades.” But we quickly discovered a blind spot: it can be surprisingly clumsy with specific, literal terms.

Imagine a user searching for a product named “Project Firefly-7B” or a specific error code like ERR_NVLINK_FAIL. Because vector search is looking for semantic similarity, it might not find the exact document if the surrounding text doesn’t perfectly align with the query’s meaning. It might return documents about “fireflies” or “NVIDIA link errors” in general.

This is where old-school technology makes a triumphant comeback. Traditional keyword search algorithms, like BM25, are exceptionally good at this. They don’t care about meaning; they care about matching exact terms.

The modern solution is Hybrid Search: we run both searches in parallel. The query is sent to our vector database for a semantic search, AND it’s sent to a keyword-based search engine (like Elasticsearch or OpenSearch) for a literal search. We then take both sets of results, merge them using a ranking algorithm, and use this combined, superior list as the context for our LLM.

B. The “Lost in the Middle” Problem: LLM Psychology 101

This last one is a fascinating and non-intuitive quirk that was discovered by researchers at Stanford, UC Berkeley, and Samaya AI in 2023. They found that LLMs don’t pay equal attention to all parts of their context window. Much like humans listening to a long list, they have a strong primacy and recency bias. They remember information presented at the very beginning and the very end of the prompt far better than the information “lost in the middle.”

Knowing this has immediate, practical implications for our RAG pipeline. If we just dump our 5 retrieved chunks into the prompt one after another, the chunk in the 3rd position might be partially ignored by the model, even if it’s highly relevant.

The engineering fix is simple but effective prompt management. After using our re-ranker to identify the single most important chunk, we don’t place it randomly. We deliberately engineer the prompt to place it at the most valuable real estate: the very end of the context block, just before the user’s question. This ensures this critical piece of information has the LLM’s full attention right as it begins to formulate an answer. It’s a small change that acknowledges we aren’t just programming a machine; we’re working with the cognitive biases of a new kind of intelligence.

Where to Go From Here: Your Turn to Build

Reading about a system is one thing; building it is another. If you’re ready to get your hands dirty or dive deeper into the research, here are some practical next steps.

For the Builder:

1. Start with a Framework: Don’t reinvent the wheel. Frameworks like “LangChain” and “LlamaIndex” provide the essential plumbing to connect all the components of a RAG system, from data loaders to vector databases to LLMs.

2. Run a Local PoC: You don’t need a massive cloud setup to start. Use an in-process vector database like ChromaDB or Meta AI’s FAISS library to build a complete RAG pipeline on your own machine.

3. Choose Your Embedding Model Wisely: The quality of your retrieval depends heavily on your embedding model. The Hugging Face MTEB (Massive Text Embedding Benchmark) leaderboard is the definitive resource for comparing the performance of hundreds of available models.

For the Researcher:

1. The Foundational Paper: “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”: This is the paper that started it all. It lays out the core “RAG-Sequence” and “RAG-Token” architectures and provides the intellectual foundation for the entire field. It’s the essential first read to understand the original vision.

2. Understanding LLM Limitations: “Lost in the Middle: How Language Models Use Long Contexts”: A crucial reality check. It empirically demonstrates that LLMs don’t pay equal attention to all parts of their context window, with performance degrading significantly when facts are in the middle of a long prompt.

3. The Evaluation Framework: “RAGAS: Automated Evaluation of Retrieval Augmented Generation”: If you can’t measure it, you can’t improve it. RAGAS provides a framework for evaluating the different components of a RAG pipeline (retriever, generator) without needing human-annotated ground truth answers. It introduces key metrics like faithfulness and context_relevancy that are now industry standards.

4. The Self-Aware System: “Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection”: The shift from a static RAG pipeline to an adaptive, intelligent one. Self-RAG trains an LLM to decide for itself whether it needs to retrieve information and to critique the quality of the retrieved documents before answering, a major step towards more autonomous systems.

5. Fixing Bad Retrieval: “Corrective Retrieval Augmented Generation (CRAG)”: Tackles one of RAG’s most common failure points: what to do when the retriever returns irrelevant documents. CRAG introduces a lightweight retrieval evaluator to assess document quality and triggers web searches to supplement the knowledge base if the initial retrieval is poor, making the system more robust.

The real work of building useful, fact-based AI systems is just getting started. RAG provides the foundation. Now, go build one. :)