Should we choose RAG or fine-tuning?

RAG fits use cases where information changes frequently and you need to cite a source — internal policy lookup or support-desk Q&A, where the underlying documents are updated constantly and you must be able to show why an answer was given. Fine-tuning fits cases where you want to change the model's behavior itself: tone, phrasing, domain-specific style. The two are not mutually exclusive — in practice it's common to fine-tune a model's behavior and then use RAG on top of it to inject current information.

Does adopting RAG guarantee accurate answers from an internal bot?

No. RAG can only be as accurate as the documents it retrieves — if retrieval quality is poor, the system will generate a plausible-sounding answer grounded in the wrong document. That's arguably a worse failure mode than an obvious error, because a fluent, confident answer makes it hard to notice retrieval went wrong. Accuracy usually hinges less on model quality than on how well the source documents are maintained and how the retrieval pipeline (chunking, embeddings, reranking) is designed.

Can a small or midsize company realistically build RAG?

The technical barrier keeps falling — managed cloud services can get a working prototype running in a matter of weeks. The real determinant of success isn't the technology, though; it's how well-organized the source documents are. If your operational knowledge is scattered across paper, spreadsheets, and individual employees' heads, RAG has nothing to retrieve from and won't function no matter how well it's built.

What is RAG (retrieval-augmented generation)? How it puts your data to work, and where it breaks

RAG is a technique where an AI system searches your own documents before answering a question, then generates its response grounded in what it finds. Unlike fine-tuning, which rebuilds the model itself, RAG queries an external knowledge source at answer time — this distinction is the core of what makes it useful. Most “chatbots that answer from our internal manual” that companies build today run on this pattern.

RAG (Retrieval-Augmented Generation) has become the default architecture for connecting generative AI to enterprise data. But the expectation that “installing RAG solves internal knowledge search” often collides with reality: teams that actually deploy it report it doesn’t behave the way they hoped. This piece defines RAG, walks through how it works mechanically, covers what actually determines its accuracy, and lays out what to verify before adopting it. Understanding RAG as a piece of technical literacy is foundational to not misjudging AI investment decisions.

What RAG is: how it differs from fine-tuning and long-context prompting

There are three broad ways to give an LLM access to company-specific information.

Approach	What it does	Best suited for	Weakness
RAG	Retrieves documents relevant to the question and feeds them into the prompt at generation time	Frequently updated information; use cases that require citing a source	Quality is bottlenecked by retrieval accuracy
Fine-tuning	Retrains the model’s own parameters on additional data	Adjusting tone, domain vocabulary, or behavior	Costly to keep current; every update requires retraining
Long-context prompting	Pastes all relevant documents directly into the prompt	Small, bounded sets of documents	Cost, latency, and accuracy all degrade as document volume grows

Of the three, RAG is the one enterprises reach for most often, and the reason is straightforward. Internal policies, manuals, and meeting notes change constantly, and the retraining cost of keeping a fine-tuned model current doesn’t pencil out. For workflows where you need to verify why an answer was given — compliance checks, customer support with an audit trail — RAG’s ability to point to the exact source document is a practical advantage.

The common misconception worth correcting: RAG is not a technique that makes a model “smarter.” It doesn’t add knowledge to the model itself. Instead, every time a question comes in, it fetches relevant information from an external source and instructs the model, “answer within the scope of this material.” Missing this distinction is the root of most “it doesn’t work as expected” complaints, which we cover below.

How it works: four steps from chunking to generation

RAG’s internal pipeline breaks down into four steps.

Figure 1: The four steps of RAG. Steps 1–2 happen ahead of time; steps 3–4 run at query time.

Step 1: Chunking. Feeding an entire policy manual into search as one block means irrelevant sections get pulled in alongside relevant ones. So documents are split into fragments — typically a few hundred to a thousand characters — along meaningful boundaries. How you chunk is the first decision point that determines downstream accuracy.

Step 2: Embedding. Each fragment is converted into a vector — a numerical representation of its meaning. Terms that are phrased differently but mean the same thing (“PTO” and “paid time off,” say) end up close together in vector space. These vectors are stored in a vector database.

Step 3: Retrieval. The user’s question is embedded the same way, and the system pulls the top handful of fragments whose vectors are closest to it. This retrieval step is where the technique gets its name.

Step 4: Generation. The retrieved fragments are passed to the LLM along with an instruction along the lines of “answer using this material,” and the model generates a response. The intent is for the model to answer within the scope of what it was given, not from whatever it happened to memorize during training.

Four factors that determine accuracy

RAG’s accuracy in practice depends far more on the surrounding pipeline than on the underlying model. In rough order of how often each one causes real problems:

1. Data hygiene: is there actually something worth retrieving?

RAG can only retrieve documents that exist. If your latest policy and an outdated version are both floating around, terminology varies by department, or a rule only exists as something people say out loud, there’s no correct document for the system to find — retrieval quality is a moot point. We cover this underlying challenge in more depth in Where should a small or midsize company start with AI adoption? (Japanese).

2. Chunking: granularity and where you cut

Chunks that are too large pull in irrelevant context; chunks that are too small lose it. If the rule for PTO carryover sits at the end of a clause while its preconditions are stated at the start, splitting mechanically by character count can sever the two — and the precondition never surfaces in a retrieved fragment. Chunking that respects heading structure and paragraph boundaries avoids this.

3. Retrieval quality: “semantically close” is not the same as “contains the answer”

Vector search finds text that’s semantically similar to the query — it does not find text that contains the correct answer. A question about PTO carryover can easily surface a fragment about how to file a PTO request, which is semantically adjacent but doesn’t answer the question. Two mitigations are common: reranking, which re-scores retrieved candidates before generation, and hybrid search, which combines keyword search with vector search.

4. Evaluation: are you actually measuring accuracy?

Running on vibes (“it seems to work okay”) means you won’t notice when accuracy degrades. A proper setup maintains a test set of representative questions paired with their correct source documents, and regularly measures recall (does the correct document show up in the retrieved set?) and end-to-end answer accuracy. Swapping in a newer model without an evaluation framework in place tells you nothing about whether perceived quality actually improved.

Why internal chatbots built on RAG so often disappoint

Companies that deploy a RAG-powered internal Q&A bot tend to report a small, recurring set of complaints.

Answer quality varies wildly depending on how the question is phrased. This is usually a retrieval-stage problem. “PTO carryover” and “how long can I keep unused vacation days” can surface entirely different fragments even though they’re asking the same thing. Retrieval design that absorbs phrasing variation — or pre-registering likely paraphrases — helps.

The bot cites outdated information. Updating the source documents doesn’t automatically update what the system retrieves; the embedding step has to be rerun for changes to take effect. Many deployments are missing the operational link between “someone edited the policy” and “the vector database got rebuilt.”

The bot produces confident, plausible, wrong answers (hallucination). When retrieval surfaces fragments that don’t actually address the question, the LLM will sometimes force an answer out of them anyway. Getting the system to say “I couldn’t find relevant information” instead requires prompt design that halts generation when retrieval relevance falls below a threshold.

The source documents don’t exist in a searchable form at all. This is a problem that predates RAG entirely — if operational knowledge only lives in an old spreadsheet or on paper, there’s nothing for RAG to search.

A checklist before adopting RAG

Before committing to a RAG build, it’s worth confirming the following:

Does the information you want to make searchable actually exist as a document — or does it only live as oral tradition or in someone’s head?
Do you have a handle on document versioning and update frequency, and who owns rebuilding the vector database each time something changes?
Can you design the system to say “I don’t know” rather than force an answer?
Do you have (or can you build) a test set of representative questions paired with correct source documents, to measure accuracy quantitatively?
Who will own ongoing tuning — chunking strategy, retrieval method — after launch?

Skipping the first two of these is the most common way a RAG deployment ends up technically functional but practically useless: it runs correctly and still doesn’t return the answers people expect. RAG is a technology that has to be evaluated alongside data readiness, not in isolation — a theme we also explore from the cost and scaling side in Architectural decision axes for embedding LLMs into a product (Japanese). If you’re weighing the operational cost of running a vector database, the framing in Database basics for business decisions (Japanese) is relevant too.

Figuring out where your own data readiness stands, and where to start, is often easier with someone walking the floor with your team and looking at the actual documents and workflows.