Adding AI to a Business App Without the Hype: A RAG Playbook
Retrieval-augmented generation is the most practical way to put AI inside an ERP or SaaS product. Here's the playbook I use, from chunking to evaluation.
Most business teams don't need a chatbot that knows everything. They need one that knows their policies, contracts, SOPs and product docs — and admits when it doesn't. That is exactly what retrieval-augmented generation (RAG) is for.
The shape of a RAG system
RAG has two halves. At ingest time you split documents into chunks, turn each chunk into an embedding and store it. At question time you embed the question, retrieve the closest chunks and give them to the model as context.
- Ingest: load → clean → chunk → embed → store
- Answer: embed question → retrieve → re-rank → generate with citations
Everything that goes wrong in production goes wrong in one of those arrows.
Chunking decides your quality ceiling
If the right paragraph never comes back from retrieval, no model can answer well. Chunk along the document's own structure — headings, sections, table rows — rather than fixed character counts.
- 01Keep chunks between roughly 300 and 800 tokens
- 02Overlap neighbouring chunks slightly so sentences aren't cut in half
- 03Store the document title and section heading with every chunk
- 04Keep a pointer back to the source so you can cite it
Bad answers are usually retrieval bugs wearing a language-model costume.
Permissions belong in retrieval, not in the prompt
In business software, not everyone may read everything. Never retrieve a chunk and then ask the model to "ignore anything the user shouldn't see". Filter by tenant, team and role inside the vector query so forbidden text never reaches the model at all.
This is the same rule as any API: authorisation happens on the server, before data leaves the database.
Make every answer verifiable
Ask the model to answer only from the provided context and to cite the chunks it used. Render those citations as links to the source document. Users trust an assistant they can check, and support teams can see exactly why an answer was wrong.
When to say "I don't know"
If the best retrieval score is below a threshold, skip generation and say the answer isn't in the knowledge base. A confident wrong answer about a refund policy costs far more than an honest gap.
Evaluate before and after every change
Build a small set of real questions with known correct sources — fifty is enough to start. Run it whenever you change chunking, embeddings, prompts or models, and track:
- Whether the right source was retrieved
- Whether the answer was correct and cited
- Latency and cost per question
Start narrow
The best first project is one well-maintained document set with a clear owner — HR policies, or product documentation. Prove accuracy there, give admins a console to manage sources, then expand. AI features earn trust the same way every other feature does: by being reliably right in a small scope first.
Building something that has to scale?
I help teams design architecture, tighten performance and ship products that keep working as the business grows.
