Free guide · building a customer-service assistant
Build a support assistant you can actually trust
Everyone will tell you to point an AI at your support inbox. Do that and you get an assistant that answers every question with total confidence and no way for you to tell which answers are right. It will quote a lead time you stopped honouring, a discount you no longer offer, and a product you stopped selling — each with a real past email as its source.
This is the method for building one that knows what it knows. It runs on your own inbox history and any capable AI model. The assistant is the last step. What comes first is counting what your customers actually ask, and deciding what is true.
What is actually in a support archive
Stark Side Gear sells rooftop-tent mounting hardware. Its inbox held about a thousand records and roughly 1,600 customer questions. Here is what an assistant would have learned from them, unfiltered.
questions about a product the company stopped selling
more than 240questions about a product the company stopped selling
the discount its own past replies quote. The live rate is 5%
10%the discount its own past replies quote. The live rate is 5%
a summary inverted which mounts are hard to reach — the largest single driver of returns
front / reara summary inverted which mounts are hard to reach — the largest single driver of returns
None of that is a retrieval problem, and no better model fixes it. The archive is a record of what was said, including everything that has since stopped being true. An assistant pointed at it answers confidently and wrong, cites a real past email as its source, and gives you no way to tell the good answers from the bad.
The same count tells you what the assistant has to get right. Compatibility alone was over 40% of everything asked — larger than the next three together, and nobody predicts that from memory:
compatibility
over 40%
other
which product
installation
shipping
returns
security
pricing
Bigger than the next three combined — and nobody predicts that from memory.
The outcome
What you end up with
Four things exist at the end that do not exist now.
An assistant that drafts replies with its sources attached. Every draft cites the approved answer it came from, so the person on the desk edits and sends instead of starting from a blank box — and can see in one glance whether to trust it.
A written rule for when it stops. Order status and stock route to the live system or return nothing. Anything your approved sources do not cover comes back unresolved rather than invented. Knowing when it abstains is what makes the rest of it usable.
One approved record per fact, with a source and an owner. Lead time, warranty window, restock policy — each in one place that every answer cites, instead of scattered across six surfaces that drift apart.
A ranked list of what your customers actually ask. Counted from your own export, not guessed. Most businesses are wrong about their own top three — and the same list rebuilds your public FAQ as a by-product.
What it takesYour expert’s time goes up before it goes down — pulling the export, ruling on facts one at a time, reviewing drafts. You need model access: a paid AI subscription for the conversational work, and API credits for the passes over your archive, which is where the spend lands. Partially-redacted customer text leaves your building to reach the model. And human review of policy, pricing, warranty and safety never ends. The full list of limits →
Fit
Does this apply to you
If three of these five are true, it does.
One topic eats your inbox and you have never counted which.
The correct answer lives in a person, not a document. Your best tech knows the spec. Nothing signed says it.
Your public FAQ and your internal files disagree, and you cannot tell which is right without asking someone.
One fact lives on six surfaces, and changing it means finding all six by hand.
You want AI on the support desk and cannot state what it may commit to. Refund? Reship? A delivery date? If you cannot say it in one sentence, the model decides for you.
The method
The method
Eight stations, each with a gate — the condition that has to be true before the next one starts. Open any station for what to run, the prompts, and what to tell an assistant. Seven are mechanical. Station 5 is the one that decides whether any of it is trustworthy.
1Raw historynot quotableEvery email, chat and review. Curation input. Never an answer.
↓ scrub, count, cluster
2Curationnot quotableCandidate facts, conflicts, and what nobody has decided yet.
↓ a person signs — nothing moves without this
3Approved answersquotableOwned, signed, dated. The only layer an assistant may quote.
↓ generates
4Search indexdisposableGenerated. Never hand-edited. Safe to delete and rebuild.
Four layers, one quotable. Everything turns on the arrow between 2 and 3 — a person signing. Skip it and you have pointed a search tool at your history, which is where the confident wrong answers come from.
Prepare
Mechanical. Get the material safe to work on.
01
Decide five things
Who signs, what the assistant may never decide, what data may be held, and what happens when a source is silent.ownerowner
None of these are about software, and all of them are expensive to change once customer data has moved.
Who signs? One named person whose approval makes a statement citable. Not a team. Reviewers who are not approvers get labelled as reviewers, or their comments get treated as sign-off.
What can the assistant never decide? Write the list. Anything on it gets drafted by the assistant and sent by a human.
What must come from a live system? Order status, stock, current price. Never from stored text, however recently built.
What is the governing source for each class of fact, and what happens when it is silent? Silence blocks. It does not default.
What may never be established by repetition? Corroboration can promote an observation about how something behaves in the field. It can never establish company position.
Gate — all true before the next station
A named owner and a named approver, written down
A written rule for what data may be held, where, and for how long
A decision on which account and terms the processing runs under
What to tell an assistant
What it does
Nothing yet. Ask the five questions, record the answers verbatim, and stop.
What it must ask you
Who owns this and keeps it current? A named person.
Who approves policy, pricing, warranty and safety? If two names come back, ask which one wins.
What customer data may be held, where, and for how long?
Which account and terms is this running under?
What it must never do
Proceed before an owner and an approver are named.
Accept a department as an owner. Shared ownership is absent ownership.
Offer to make any of these decisions on their behalf.
02
Set up the workspace
Four folders, and the ignore rules written before anything is copied in.one-timeone-time
The folder layout is the governance model made physical. Raw evidence restricted, curation where judgment happens, approved answers the only quotable layer, the index generated and disposable.
kb/
├── raw/ <- the export. RESTRICTED. never leaves this folder
├── scrubbed/ <- redacted working copy. everything downstream reads this
├── curation/ <- candidates, conflicts, decisions
├── approved/ <- the only quotable layer
├── taxonomy/ <- categories, part names, customer wording
├── index/ <- generated. never hand-edited
└── evals/ <- the test questions
Why the ignore rules come firstOne build tracked an unredacted customer corpus in a private repo from the first commit and did not notice for months. Once it is in history it is in every clone and every backup. Two minutes now, or a bad week later.
Gate — all true before the next station
Your agent runs and can read the folder
raw/ and scrubbed/ are ignored before the export is copied in
The export is in raw/ and nowhere else
What to tell an assistant
What it does
Create the structure. Write the ignore rules BEFORE anything is copied in, then test that they work.
What it must ask you
Where does the export live now, and has it been copied anywhere else already?
What it must never do
Copy, move or read the export before the ignore rules are verified.
Assume an ignore file works because it exists. Test it.
Put the raw layer anywhere that syncs or backs up without being told to.
03
Redact the export
Deterministic, auditable, and it must prove its own coverage before you continue.mechanicalmechanical
A model asked to redact will miss records quietly, and coverage cannot be proven afterwards — which is exactly the property you need here. A pattern-based scrubber is auditable: you can read every rule and say what it does and does not catch.
What pattern-based redaction does not catchPersonal names in free text. “Thanks, Dave” survives — there is no reliable pattern for a first name. Bare city names survive too, deliberately, because masking them wrecks the text for counting. The scrubbed copy is reduced-risk, not anonymous. It stays internal.
Gate — all true before the next station
The leak report is clean for both emails and phone numbers
Two copies exist: an untouched archive, and the scrubbed working copy
Every file was handled — nothing silently skipped as an unrecognized type
What to tell an assistant
What it does
Run the script and read its report back. You are the operator here, not the redactor.
What it must ask you
Which email domains and phone numbers belong to the business, so they are preserved rather than masked?
What format is the export in? Mail-archive formats have to be converted first.
What it must never do
Redact by reading the text yourself.
Proceed if the leak report is not clean on every category.
Describe the scrubbed copy as anonymous.
Skip an unrecognized file type without reporting it loudly. Anything skipped is unredacted.
Establish what is true
Where the judgement lives, and the only part you cannot delegate.
04
Count what customers actually ask
Extract verbatim, cluster by need, rank by volume times what a wrong answer costs.mechanical + a second readermechanical + a second reader
Do not start by writing answers. Start by finding which questions are worth answering — demand is heavily skewed, and at the reference build one category was over 40% of everything.
Rank by two things multiplied: how often it comes up, and what a wrong answer costs. A rare question that scraps a part outranks a common one that does not.
Read every conversation in scrubbed/ and extract the questions
customers actually asked, in their own wording. One row each:
question | their wording, verbatim
intent | what they were really trying to find out
answer | what we told them, if we did
source | file and record id
Write to taxonomy/questions.jsonl. Do not summarize, interpret
or merge yet. I want the raw distribution first.
Read taxonomy/questions.jsonl and group into recurring issues.
- Group by what the customer needed, not how they worded it
- Keep every distinct phrasing as a variant of its cluster
- Flag clusters where our past answers DISAGREE with each other
- Flag clusters about parts or policies that no longer exist
Sort by volume. Give me the disagreement flags separately.
The disagreement list is the valuable outputClusters where your own past answers contradict each other are where the policy was never written down, or changed and nobody propagated it. That list is worth more than the volume ranking.
Gate — all true before the next station
A ranked list of 20–30 real intents, in customer wording
A second person has read it for what is missing
Clusters touching discontinued parts or superseded policy are marked, not deleted
What to tell an assistant
What it does
Extract questions verbatim, one row each. Then cluster by what the customer needed. Produce a ranked list and a separate disagreement list.
What it must ask you
Which of these clusters touch products or policies that no longer exist?
What it must never do
Summarize or merge on the first pass. Extract verbatim first, cluster second.
Group by how the business is organized. Group by what the customer needed.
Discard clusters where past answers contradict each other — surface them.
Rank by volume alone. Weight it by what a wrong answer costs.
05
Write down what is true
Roughly thirty statements, each with its exceptions. This is the one that cannot be delegated.the owner — the constraintthe owner — the constraint
Write down the roughly thirty statements that must never be wrong — the return window, the standard lead time, the warranty terms, the tolerance you actually hold — and beside each, its exceptions.
The exceptions are the whole job. At the reference build, the most load-bearing rule in the entire return policy — that nothing ships out before the original comes back and is inspected — appeared in no file and on no page anywhere, until the owner said it out loud in a single sentence.
statement: Returns are accepted within 30 days, and require
authorization before anything ships back.
exceptions: - International orders are final sale.
- Restocking fee 10% unopened, 20% opened.
- Nothing ships out ahead of the return coming back.
- Customer-modified parts are not accepted.
risk_class: policy
approved: <-- stays empty until a person signs it
Why this station cannot be handed to a modelA governing statement cannot be validated by the process that consumes it. If a statement is wrong, every answer consistent with it gets cleared confidently and calibration will not detect the error. This is the one input no downstream check can catch, which is why the owner is permanently on the critical path.
Gate — all true before the next station
Roughly thirty statements written, each with its exceptions
Every statement carries a risk class assigned by its source
A person has signed them — an unsigned set is inert
Topics with no authority are on an explicit unresolved list, not quietly guessed
What to tell an assistant
What it does
Interview the owner and draft candidates from their answers. Everything you draft stays marked draft with an empty approver until they sign.
What it must ask you
For each high-volume cluster: what is the rule, and what are its exceptions?
When does the normal answer stop applying?
What would a new hire get wrong here?
Which of these has never actually been written down anywhere?
What it must never do
Write a governing statement from the history. History is evidence; the owner is the authority.
Fill a gap the owner left. If they do not answer, it goes on the unresolved list.
Drop an exception to make a statement read more cleanly.
Mark anything approved. You cannot approve.
06
Check the history against it
Four verdicts. Only one clears, and only outside the high-risk classes.mechanical + calibrationmechanical + calibration
Now the volume gets processed. Each past answer is checked against the signed statements and sorted into exactly four verdicts.
Judged against this signed statement: Fits bars 0.75″–2.1″ tall. Exception: 1.6″–2.1″ needs the included spacers. Over 2.1″ does not fit.
supportedclears
“a 1.25″ bar works” — Agrees. No exception applies.
incompleteheld
“a 1.8″ bar fits fine” — True — and drops the spacer exception.
contradictedheld
“2.5″ bars are fine” — Above the maximum. Disagrees outright.
uncoveredabstain
“aero bars are fine” — No signed source mentions aero bars.
Verdict
A past answer
Why
Outcome
supported
“a 1.25″ bar works”
Agrees. No exception applies.
clears
incomplete
“a 1.8″ bar fits fine”
True — and drops the spacer exception.
held
contradicted
“2.5″ bars are fine”
Above the maximum. Disagrees outright.
held
uncovered
“aero bars are fine”
No signed source mentions aero bars.
abstain
incomplete is the dangerous one: the reply is true and still wrong, because it sends a customer away without the spacers they need.
supported agrees with a statement and its exceptions -> clears
incomplete agrees but drops an exception -> hold, fix
contradicted disagrees with a statement -> hold, review
uncovered no citable source addresses this -> unresolved list
Take 30 items the automatic check cleared.
Have the owner grade each one blind.
Compare. If you disagree on more than a handful, the statements
are ambiguous -- fix the statements, not the checker -- and re-run.
If they still disagree, the model is not good enough for this
step. Try another. That is the whole model-selection process:
measure it on your own material.
Calibrate on entailment, not on tasteOne build shelved a working checker because it scored 0.55 against an arbitrary 0.7 threshold — on a sample of ten, grading tone rather than whether a claim followed from a rule. Ten items cannot demonstrate anything either way, and tone is not what makes an answer wrong.
Gate — all true before the next station
A calibration sample the owner graded, agreeing closely enough to trust the rest
Every high-risk verdict routed to a person regardless of the checker's output
The uncovered list handed back to the owner as new statements to write
What to tell an assistant
What it does
Assign exactly one of the four verdicts to each historical answer. Report the distribution.
What it must ask you
Grade this calibration sample blind, then tell me where we disagreed.
What it must never do
Clear anything in policy, pricing, warranty or safety, whatever the verdict says.
Invent a fifth verdict, or soften contradicted into partially supported.
Scale up before the calibration sample agrees.
Report agreement rates you have not measured.
Operate
Build it, then run it with a person in the loop.
07
Build the answers and the index
One article per reusable issue. The index is generated and never hand-edited.mechanicalmechanical
One article per reusable issue — not one per email. The article is the answer that gets reused; the original conversation stays behind it as evidence. The index is generated from approved articles and disposable. When an answer changes you edit the article and rebuild.
# The customer issue
## Applies when
## The answer
## What to collect first
## Exceptions and limits
## Escalate when
## Sources
Never hand-edit the indexOne build corrected a discount rate directly inside the generated index. The source kept the old number, the drift check compared file hashes and saw nothing wrong, and it reported clean for 73 days while the wrong figure went to customers.
Gate — all true before the next station
Every indexed article is approved and cites its source
The index rebuilds from scratch, reproducibly
Nothing archived, expired or unapproved is retrievable
What to tell an assistant
What it does
Write one article per reusable issue, from approved statements only. Generate the index. Report counts against the previous build.
What it must ask you
Which candidate articles is the approver signing, and which are waiting?
What it must never do
Write an article for an issue with no approved statement behind it.
Edit the generated index for any reason, including to fix something you know is wrong.
Include archived, expired or unapproved material.
Write a reduced index and call it complete.
08
Draft, review, send
Prove it offline against real questions first. Then a person reads every reply.normal support worknormal support work
The assistant pulls the thread, the approved answer, that customer’s own order record, and the rules about what it may say — then writes a draft with its sources attached. A person reads it, edits if needed, and sends. Before any of that, prove it offline.
50-100 questions, taken from Station 4's actual customer wording.
Include, deliberately:
- ordinary phrasing, and typos
- exact part and model numbers
- ambiguous questions
- questions with NO supported answer <- the important ones
- advice that has since been superseded
- anything restricted or internal-only
The habit that decides whether this lastsWhen an article turns out wrong or thin, fix it or flag it right then, while the ticket is open and you still remember why. That one habit is the difference between a library that stays alive and one that quietly rots into another thing nobody trusts.
Gate — all true before the next station
A 50–100 question test set built from real demand, not invented
Abstention verified on unsupported questions, especially high-risk ones
A named person reviews every reply during the pilot
A review cadence and an owner for keeping articles current
What to tell an assistant
What it does
Draft from approved articles, attach citations, hand to a person. Test against the real question set before any of this goes live.
What it must ask you
Who is reviewing every draft during the pilot, and have they agreed to it?
What it must never do
Send anything.
Commit to an outcome — a refund, a return, a warranty decision, a delivery date, an expedite.
Answer from memory of the conversation rather than a retrieved, cited article.
Guess when the articles do not cover it.
Evidence
Why the gates exist
Every rule in this method came from something breaking. Read cold, “never edit the generated index” sounds like fussiness. It is not — it cost 73 days.
The check was working. It was checking the wrong thing.
A generated index entry was hand-corrected to 5% while its source document kept the old 10%. Both sides hashed the same untouched file, so the drift check matched and reported clean every day for 73 days. Hashing a file tells you the file moved. It cannot tell you a number inside it changed, and it cannot see that a generated entry now contradicts the document it came from.
The rule this produced — Fix the source and rebuild, never edit the generated layer. If a tool lets you edit it at all, that is a defect in the tool. Station 7
Fixed is not live.
A correction moved through the authority file, the FAQ source and the publish script — all three correct — and stopped, because nobody ran the publish step. The only surface a customer could see stayed wrong for 4 days. Worse: three entries on that live page existed in no internal document at all, so the public page had quietly become a source of truth nobody was maintaining.
The rule this produced — The publish step is part of the gate, and you verify it landed rather than assuming. Station 8
A build lost a third of the index behind a green checkmark
The index builder needed a credential to fetch part of its content. Without it, the builder dropped a whole tier, exited successfully, and every automated check passed. The output was 179 entries instead of 262 — not obviously broken, just a different plausible-looking shape.
The one guard that should have caught it was a count range that did not contain the correct value either, so it warned on every good build and got ignored as noise. A check that cries wolf on correct runs cannot catch the broken one.
The rule this produced — A partial result is a failure, not a smaller success. Compare counts against the last run and stop rather than shipping a reduced output. Station 7
Four more, less dramatic and just as real
Machine summaries invert details
A summary in the evidence layer said the rear mounting points were the hard ones to reach. The owner corrected it — the front ones are, and that distinction was the single largest driver of returns on the product. The records were machine-generated summaries, so the inversion was an artifact rather than a real historical answer. Every other directional or dimensional claim in the same corpus carries the same class of error.
Corroboration is not authority
Fifty past emails saying the same thing tells you what was said fifty times. It does not tell you it was correct, and it does not make it policy. Repetition can promote an observation about how something behaves in the field. Only someone with authority to set the position can establish the position.
Silence blocks — it does not default
A discount code appeared repeatedly in historical records and not one of them stated its percentage. The temptation is to infer one from context. A guessed number is indistinguishable from a real one once it is in the index, and it gets quoted with total confidence. Two different rates were live at the same time before this was caught.
Your live site may be more current than your internal files
An audit concluded three policies were missing — exchanges, damage on arrival, and a returns process. All three were published and live. The internal documents were the stale copy. Read what your customers can actually see before concluding a policy does not exist. Three out of three of those findings reversed on it.
The two rules all of them come back toA repeated historical answer is not policy — corroboration establishes what was said, never what is true. And silence blocks rather than defaults: where nobody wrote a number down, the answer is unresolved, not a sensible-looking guess.
By-product
The bonus: the same count rebuilds your FAQ
Station 4 produces a ranked demand map so the assistant knows what it has to answer. That map has a second use nobody plans for — it tells you exactly which public FAQ entries to write, and in what order. Every question answered on that page is one the assistant never has to draft and a person never has to check.
Two rules and a warning. Rank by counted volume, not by category logic. Use the customer’s own wording, captured verbatim at Station 4 — it is also what they type into a search box. And generate the page from the approved layer rather than hand-writing entries beside it, because a public FAQ is one more surface that can drift, and the one your customers actually read.
Why this matters more than it used toYour public FAQ is what an AI assistant quotes about you. Search engines read that page. Shopping assistants read that page. None of them call your support line, and none of them ask whether the number is current. Ungoverned, your stale answer is the one your customer gets — delivered by a model, in your voice, with you not in the loop.
Limits
What this is, and what it is not
The full version of the limits, since you have read this far. This is a method, a redaction script, and a set of plain-text prompts. There is nothing to install and no dashboard. The judgment stays yours.
What it is
A four-layer separation — raw history stays evidence, approved articles are the only quotable text, the search index is generated and disposable.
A redaction script that strips structured identifiers before your text goes anywhere.
Plain-text prompts you can read, edit, and point at any model.
A fixed verdict vocabulary. A reviewer’s call is one of four words, not a feeling.
A worked reference build on real data, including its own defects.
What it is not
Not free — you are buying model access. A paid subscription to an AI assistant covers the conversational stations. The bulk passes are different: clustering sixteen hundred questions, then checking every past answer against every statement, means processing your whole archive several times over — that is API credits, billed per token, and it is where the spend concentrates.
And not free of paperwork. Someone has to accept the vendor’s terms, and if you carry a contractual or regulatory obligation on customer data, a data-processing agreement is the thing that makes this allowed rather than merely possible. That is a conversation to have before the export moves, not after.
Not local. Classification and drafting are model calls. Redaction catches structured identifiers — not a customer’s name typed mid-sentence.
Not enforced. The gates are conventions. “Approved” is a word somebody types. Nothing blocks a build or refuses a publish.
Not a phase you graduate from. Policy, pricing, warranty and safety stay under human review permanently.
Not a live-system tool. Order status and stock need the system that owns them.
Not a substitute for deciding. It will not fix a business that never wrote its policies down. It will show you that clearly, which is not the same thing.
What the assistant is never allowed to decide
Never
What it may do instead
Approve a return or a refund
State the policy and the window, list what the customer must provide, route it
Issue an authorization number
Explain that authorization is required and how it is requested
Decide a warranty or quality claim
State the coverage, the exclusions, and what evidence a claim needs
Commit to a date or an expedite
State the standard lead time, then route to whoever can commit
Quote a current price or quantity
Read it live from the system of record, or say it cannot and route
Send anything by itself
Draft, cite its sources, and wait for a person
The general ruleAn assistant may describe what is already written down, collect what is missing, and route. It may never commit the business to an outcome. If a sentence would oblige you to do something, a person writes it.
The working kit
Everything above is yours to use — the method, the gates, and the machine-readable version for your assistant. The kit is the part you cannot get by reading. Files that run, configured for your export.
.pyThe redaction scriptDeterministic, auditable, and it re-reads its own output and refuses to pass if anything leaked.
shThe folder scaffoldThe four layers, with the ignore rules already written so the raw layer can't get committed.
.mdThe prompt packEvery station's prompt as a file you drop in the folder, rather than copy one at a time.
▦The workbookYour progress, gates and notes — saved in the file itself, so it survives a refresh and moves between machines.