The first thing everyone learns about RAG:
"Put your documents in a vector database, retrieve the relevant chunks, hand them to the LLM."
Simple enough. Then a new component shows up in every production architecture diagram: the reranker.
And a fair question follows. If the vector database already found the most relevant documents, why is there another ranking step?
Think about hiring
Say you're hiring an engineer and 10,000 resumes land in your inbox. Nobody sane has the hiring manager read all of them.
So a recruiter does a first pass and shortlists 50 candidates. That's retrieval.
Then the hiring manager goes through those 50 properly and picks the top 5. That's reranking.
Two stages, two different jobs. The recruiter is fast and broad. The hiring manager is slow and careful.
Step 1: Vector retrieval
A user asks a question. We convert the query into an embedding, and the vector database finds documents that are semantically close to it. (I covered the mechanics in What Is an Embedding in AI and What Is a Vector Database if you want the background.)
Take this query:
"How do I reset my AWS password?"
The retriever might return:
- AWS IAM User Guide
- AWS Security Best Practices
- AWS Account Billing
- Password Policy Documentation
- AWS Login Documentation
Look closely. Every one of these is about AWS. Every one touches accounts, security, or authentication. And they're not equally useful for answering the question.
The vector database did exactly what it was built to do. It found similar information.
But similarity and relevance are not the same thing.
Where the reranker comes in
A reranker takes the original query and the retrieved documents and evaluates them together.
The vector search effectively asked:
"Which documents are most similar to this query?"
The reranker asks something sharper:
"Which of these documents is most relevant for answering this specific question?"
The order might become:
- IAM User Guide
- Password Policy Documentation
- AWS Security Best Practices
- AWS Login Documentation
- AWS Account Billing
Weaker results move down. Stronger ones move up. And now the LLM gets better context.
Why not just retrieve fewer documents?
Fair objection. Why not tell the vector database to return the top 3 and skip the extra step?
Because the two stages have different trade-offs. The first stage is built to be fast and broad. The second can afford to be slower and more precise, because it only looks at a small candidate set instead of your whole corpus.
So the common architecture looks like this:
User Query ↓ Vector Search ↓ Top 20 documents ↓ Reranker ↓ Top 5 documents ↓ LLM ↓ Answer
The retriever gives you candidates. The reranker fixes the ordering.
Why context quality matters
An LLM doesn't magically know which retrieved chunk is the important one. You hand it a context window and ask it to reason over whatever is in there.
If that context has irrelevant documents, contradictory information, duplicates, or noisy chunks, the answer suffers. Every time.
Better retrieval means better input. Better input gives the model a real chance of producing something useful.
This is why RAG quality isn't just about the LLM. It's about what you put in front of it. (Chunking plays a big part here too, which I wrote about in Chunking Strategies — Why Getting This Wrong Quietly Kills Your RAG.)
The bigger lesson
Don't stop at "my vector search returns documents."
Ask harder questions:
- Are the right documents being retrieved?
- Are the most useful chunks ranked first?
- Are irrelevant chunks reaching the LLM?
- How does retrieval quality change across different queries?
- Can we actually measure retrieval performance?
- Does reranking improve answer quality?
Once you start asking those, you're no longer building a RAG demo. You're engineering a retrieval system.
That's where production RAG gets interesting.
The mental model
Vector Database — find good candidates quickly.
Reranker — pick the best of them.
LLM — use those candidates to write the answer.
Better retrieval, better context, better answers.
— Cheers, NP