For most enterprise document use cases, retrieval-augmented generation (RAG) is the right default: it answers from your current documents, cites the page behind each statement and respects who is allowed to see what. Fine-tuning is better suited to teaching a model a format, tone or narrow task than to teaching it facts, and long-context prompting works for small, fixed document sets. This guide, written from a RAG consulting perspective, explains when each approach fits, why auditability matters, where graph RAG helps, and how to evaluate the result.
The three approaches in brief
Retrieval-augmented generation (RAG)
RAG splits your documents into passages, indexes them for search (usually a mix of semantic embeddings and keyword search), and at question time retrieves the most relevant passages and gives them to the model along with the question. The model writes an answer grounded in those passages and can point to exactly which ones it used. The model itself does not change; the knowledge lives in the index.
Fine-tuning
Fine-tuning continues training a model on your examples so that its weights change. It is effective for teaching consistent behaviour: always output this JSON structure, classify tickets into these categories, write in this house style, use this specialized vocabulary. It is a weak way to teach facts. The model absorbs patterns from the training data, but it cannot reliably tell you which document a fact came from, and it may blend or misremember details.
Long-context prompting
Modern models accept very long inputs, so a tempting option is to paste entire documents into the prompt. For a handful of contracts or a single long report, this works well and needs no infrastructure. It scales poorly: cost and latency rise with every token, models can overlook details in the middle of long inputs, and you still need a way to decide which documents to include for a given user and question.
When each one fits
| Requirement | RAG | Fine-tuning | Long context |
|---|---|---|---|
| Answers from a large, changing document set | Strong | Weak | Weak beyond small sets |
| Citations to source pages | Native | Not reliable | Possible, harder to verify |
| Per-user access control | Enforced at retrieval | Not possible within the model | Must filter before prompting |
| Removing a document | Delete from index, effective immediately | Requires retraining | Stop including it |
| Consistent output format or style | Via prompting | Strong | Via prompting |
| Setup effort | Moderate: ingestion, indexing, evaluation | High: curated training data, training runs | Low |
| Cost per question | Moderate | Low once trained | High for large inputs |
The approaches are not mutually exclusive. A common production design uses RAG for knowledge, prompting or a small fine-tune for output format, and long context to give the model the full text of the two or three documents retrieval identified as most relevant.
Citations and auditability: the enterprise deciding factor
In regulated or high-stakes settings, an answer without a source is a liability. A lawyer, auditor, analyst or clinician needs to check the claim against the original before acting on it. RAG makes that possible because the system knows exactly which passages it gave the model. A well-built system can link each statement in the answer to the page it came from, so the reader can verify in seconds.
Fine-tuning works against this. Once information is baked into model weights, there is no record of where any particular statement came from. That creates three practical problems:
- You cannot enforce access control. If a model was trained on HR files, anyone who can query it may be able to coax that information out. RAG filters retrieved passages by the user's permissions before the model sees them.
- You cannot cleanly delete. If a document must be removed, whether because it is outdated, superseded or the subject of a privacy request, a RAG index can drop it immediately. A fine-tuned model has to be retrained.
- You cannot audit. When a regulator or internal reviewer asks why the system said something, RAG can show the retrieved passages and the prompt. A fine-tuned model can only show its output.
For organizations subject to privacy law, these are not edge cases. They are the core requirements.
Graph RAG for cross-document questions
Standard RAG retrieves passages that are similar to the question. That works well for "what is our parental leave policy?" It works poorly for questions whose answer is spread across many documents and depends on relationships: "which of our suppliers are named in contracts that include an unlimited liability clause?" or "how are these three entities connected?"
Graph RAG addresses this by extracting entities (organizations, people, products, contracts, dates) and the relationships between them into a knowledge graph during ingestion. At question time, the system can traverse the graph to find connected entities, then retrieve the passages that describe those connections. The result is better coverage on multi-hop and aggregate questions.
Graph RAG costs more to build and maintain. Entity extraction must be checked, and entities need to be resolved so that "ABC Corp," "ABC Corporation" and "A.B.C. Corp." are treated as one. Use it where cross-document questions are common, such as due diligence, compliance investigations, research portfolios and contract management, rather than as a default.
Building a RAG system that holds up
Most RAG failures are retrieval failures, not model failures. The model cannot cite a passage it never received. The parts that matter most are:
- Ingestion quality. Parse PDFs, scans, tables and spreadsheets properly. A table flattened into nonsense or a scanned page with no OCR is invisible to search.
- Chunking that respects structure. Split on headings and sections rather than arbitrary character counts, and keep metadata such as document title, page number, date and permissions with every chunk.
- Hybrid search. Combine semantic search with keyword search so that exact terms, such as product codes, clause numbers and names, are not lost. A re-ranking step on the top results often improves precision.
- Query planning. For complex questions, let the system break the question into several searches and combine the results. This is the agentic approach used in Akurion.ai, which plans its own searches and cites the page behind every statement.
- Refusal when unsupported. Instruct and test the system to say it cannot find an answer rather than fill gaps from the model's general knowledge.
How to evaluate
Evaluation is what separates a convincing demo from a system people can rely on. Build it before you tune anything.
- Create a question set. Gather real questions from the people who will use the system, along with correct answers and the passages that support them. Include questions with no answer in the documents.
- Measure retrieval separately. For each question, check whether the supporting passages appear in the top results. If retrieval misses, no prompt change will fix it.
- Measure generation. Score whether the answer is correct, whether every claim is supported by the retrieved text (faithfulness), and whether citations point to the right pages.
- Check refusals. Confirm the system declines questions the documents do not answer, and declines to reveal content the user is not permitted to see.
- Regression-test changes. Re-run the set every time you change chunking, embeddings, models or prompts, and track results over time.
Automated scoring with a model as judge can speed this up, but spot-check its grades against human judgement, especially early on.
A quick decision guide
If you are choosing an approach for a specific project, these questions usually settle it.
- Does the answer depend on facts in your documents? Start with RAG. Fine-tuning will not give you reliable, citable facts.
- Do different users have different access rights? You need retrieval with permission filtering. This rules out fine-tuning on the restricted content.
- Do the documents change weekly or monthly? RAG, since re-indexing a changed document takes minutes and retraining a model does not.
- Is the set small, fixed and shared by everyone? Long-context prompting may be enough, and it is the quickest to ship.
- Is the problem the shape of the output rather than its content? For example, a fixed report layout, a classification scheme or a domain's writing conventions. Try careful prompting with examples first; consider fine-tuning if prompting cannot get you to consistent results.
- Do users ask questions that connect many documents? Add graph RAG on top of standard retrieval for those question types.
Common mistakes
Three mistakes come up repeatedly. The first is fine-tuning a model on a document archive in the hope that it will "know the business," then discovering it cannot cite anything and occasionally invents details. The second is judging a RAG system on a few friendly questions chosen by the team that built it, rather than on questions gathered from the people who will use it. The third is ignoring ingestion: spending weeks comparing models while scanned pages, multi-column layouts and tables are being silently mangled before they ever reach the index. Fixing document parsing often improves answers more than changing the model.
Where to run it
If your documents contain personal or confidential information, decide where indexing and inference will run before you start. Embeddings and retrieved passages are as sensitive as the documents themselves. For many Canadian organizations, a private LLM deployment with open-weight models, hosted in Canada or on-premises, keeps the entire pipeline under their control.
How Dassjay Labs helps
Dassjay Labs is an AI consultancy and product studio in Kitchener, Ontario, serving clients across Canada. Our RAG and document AI work covers ingestion of difficult documents, hybrid and graph retrieval, permission-aware search, citation and evaluation, built on the same foundations as Akurion.ai's agent mode, graph RAG, MCP, SSO and workflow features. If you are deciding between RAG, fine-tuning or a mix, contact us and we will help you test the options on your own documents.