RAG (Retrieval-Augmented Generation) is an architecture in which an application retrieves relevant documents from a knowledge source at query time, typically via vector or hybrid search, and adds them to the prompt of a language model. The model then answers based on this context, which makes answers more current, more specific and easier to verify.
What is RAG?
RAG (Retrieval-Augmented Generation) combines a search step with text generation. Instead of relying only on what a language model learned during training, the application first looks up relevant information in its own data, such as documentation, tickets, contracts or a product catalog, and passes the results to the model as context. The term was popularized by a 2020 research paper by Lewis et al. and has since become one of the most common patterns for enterprise LLM applications.
RAG is attractive because knowledge can be updated by re-indexing documents rather than retraining a model, access controls can be applied at retrieval time, and answers can cite their sources. At the same time, it adds moving parts that need their own monitoring, which is why RAG is a core topic in LLM observability.
How RAG works
- Ingestion: documents are split into chunks, converted into embeddings by an embedding model and stored in a vector index, often together with keyword indexes and metadata such as source, date and permissions.
- Retrieval: at query time the user question is embedded and the most similar chunks are fetched, frequently with hybrid search that combines vector similarity and keyword matching.
- Reranking and filtering: a reranker or rules reorder and trim the candidates, apply access control and remove duplicates.
- Augmentation: the selected chunks are inserted into the prompt together with instructions.
- Generation: the model produces the answer, ideally with references to the chunks it used.
Variants include agentic RAG, where an agent decides when and what to retrieve, and multi-step retrieval for complex questions.
Each stage has its own tuning knobs: chunk size and overlap, the embedding model, the number of candidates retrieved, the reranker and the prompt template. Changing any of them can improve one type of question and degrade another, so changes should be tested against a fixed set of questions and observed in production.
Where RAG fails
- Retrieval misses: the right document exists but is not retrieved because of poor chunking, a weak embedding model or a query that is phrased differently from the source.
- Stale or wrong data: the index contains outdated versions or documents the user should not see.
- Context overload: too many or too long chunks drive up token usage and latency and can bury the relevant passage.
- Unfaithful generation: the model ignores or contradicts the retrieved context, which is a form of hallucination.
- Latency stacking: embedding, search, reranking and generation add up, and each can become the bottleneck.
Observing a RAG pipeline
A RAG request maps well onto a distributed trace with one span per stage. The OpenTelemetry GenAI semantic conventions (status: development) define operation names such as embeddings, retrieval and chat, and attributes such as gen_ai.data_source.id for the knowledge source, gen_ai.request.model and token counts. Retrieved documents and query text are defined as opt-in attributes, because they may contain sensitive content.
Useful signals per stage:
| Stage | Signals |
|---|---|
| Ingestion | Documents indexed, indexing lag, failed documents |
| Retrieval | Latency, number of results, similarity scores, empty results |
| Reranking | Latency, chunks kept vs. dropped |
| Generation | Input and output tokens, time to first token, finish reason |
| Quality | Context relevance, groundedness, answer relevance from evaluations |
Example: a slow RAG endpoint
An illustrative case: the p95 latency of a document assistant rises after a release. The traces show that the model span is unchanged, but the retrieval span now takes several seconds. A new metadata filter is applied after the vector search on a large candidate set instead of inside the index query. Moving the filter into the index query brings retrieval latency back to its previous level, which the trace data confirms.
Best practices
- Version your index, embedding model, chunking settings and prompt, and record these versions on each trace.
- Build a test set of questions with expected source documents and measure retrieval quality separately from answer quality, using LLM evaluation.
- Enforce permissions in retrieval, not in the prompt.
- Limit the number and size of chunks and monitor context tokens per request.
- Monitor the vector store like any other database: latency, errors, saturation and index size.
How Obseria helps
Obseria shows each RAG request as one trace from the API call through embedding, vector search and model call, using distributed tracing based on OpenTelemetry. Database and infrastructure metrics of the vector store sit next to GenAI spans and token metrics, so you can see whether a slow or poor answer comes from retrieval, the model or the data.
