Building a RAG Pipeline That Actually Cites Its Sources

It's straightforward to wire an LLM up to a vector database and call it RAG. It's much harder to build one where every answer traces back to an exact page in an exact source document — which is the actual bar for a document intelligence tool anyone will trust with real work.

Chunking is the decision that matters most

Chunk too small and you lose context — the model answers confidently from a fragment that's missing the qualifying sentence two paragraphs up. Chunk too large and retrieval gets noisy, pulling in irrelevant text alongside the relevant part, which dilutes the citation and the answer. We tuned chunk size and overlap around the actual document types users uploaded, not a generic default.

Citations have to be structural, not generated

The tempting shortcut is to ask the model to describe its own source in the answer text. That produces vague summaries like 'according to the document' — not verifiable. Instead, every chunk stored in pgvector carries its source document ID and page number as metadata, and citations are attached programmatically to whichever chunks the retrieval step actually returned, not generated by the model after the fact.

Large batches need timeout protection

Large document batches processed synchronously will hit request timeouts before embedding finishes. We moved ingestion to a background job queue, so a 200-page upload doesn't block the request thread or fail silently — it processes incrementally and the UI reflects real progress.