# Build a support assistant you can actually trust A method for building a customer-service assistant out of your own support history — one that knows what it knows, and abstains when it does not. Point an assistant at this file and it can run the method with you. Source: https://jenso.ai/resources/support-knowledge-base This file: https://jenso.ai/resources/support-knowledge-base/llms.txt Publisher: Jenso LLC (jenso.ai) Licence: free to read, quote and use, with attribution. ## The problem this solves Pointing an AI at a support inbox produces an assistant that answers every question with total confidence and no way to tell which answers are right. It will quote a lead time you stopped honouring, a discount you no longer offer, and a product you stopped selling — each with a real past email as its source. An archive is a record of what was *said*, including everything that has since stopped being true. That is not a retrieval problem and no better model fixes it. ## What a real support archive contains Measured on the reference build (Stark Side Gear, https://tentmount.com) — about a thousand records and roughly 1,600 customer questions. Volumes are given as ranges on purpose; see "About these figures" at the end. - more than 240 questions about a product the company had stopped selling - past replies quoting a 10% discount when the live rate was 5% - a machine summary that inverted which mounts are hard to reach — the largest single driver of returns Demand is heavily skewed. Counted, not guessed — largest first: - compatibility - other - which product - installation - shipping - returns - security - pricing Compatibility alone was over 40% of everything customers asked — larger than the next three combined, and nobody predicts that from memory. The ordering above is the finding; the per-category totals are not reproduced here, because the ranking is what changes what you build and the individual counts are not. ## The four layers The whole method turns on the boundary between layer 2 and layer 3: a person signing. Skip it and you have pointed a search tool at your history, which is where the confident wrong answers come from. 1. **Raw history** (not quotable) — Every email, chat and review. Curation input. Never an answer. ↓ scrub, count, cluster 2. **Curation** (not quotable) — Candidate facts, conflicts, and what nobody has decided yet. ↓ a person signs — nothing moves without this 3. **Approved answers** (quotable) — Owned, signed, dated. The only layer an assistant may quote. ↓ generates 4. **Search index** (disposable) — Generated. Never hand-edited. Safe to delete and rebuild. **Only layer 3 may ever be quoted.** ## The method — eight stations Each station has a gate: the condition that must be true before the next one starts. Seven are mechanical. Station 5 is the one that decides whether any of it is trustworthy, and it cannot be delegated to a model. ### Station 1 — Decide five things Who signs, what the assistant may never decide, what data may be held, and what happens when a source is silent. Who does it: owner None of these are about software, and all of them are expensive to change once customer data has moved. - **Who signs?** One named person whose approval makes a statement citable. Not a team. Reviewers who are not approvers get labelled as reviewers, or their comments get treated as sign-off. - **What can the assistant never decide?** Write the list. Anything on it gets drafted by the assistant and sent by a human. - **What must come from a live system?** Order status, stock, current price. Never from stored text, however recently built. - **What is the governing source for each class of fact, and what happens when it is silent?** Silence blocks. It does not default. - **What may never be established by repetition?** Corroboration can promote an observation about how something behaves in the field. It can never establish company position. **Gate — all of these must be true before the next station:** - A named owner and a named approver, written down - A written rule for what data may be held, where, and for how long - A decision on which account and terms the processing runs under **What you do here:** Nothing yet. Ask the five questions, record the answers verbatim, and stop. **What you must ask the human:** - Who owns this and keeps it current? A named person. - Who approves policy, pricing, warranty and safety? If two names come back, ask which one wins. - What customer data may be held, where, and for how long? - Which account and terms is this running under? **What you must never do:** - Proceed before an owner and an approver are named. - Accept a department as an owner. Shared ownership is absent ownership. - Offer to make any of these decisions on their behalf. ### Station 2 — Set up the workspace Four folders, and the ignore rules written before anything is copied in. Who does it: one-time The folder layout is the governance model made physical. Raw evidence restricted, curation where judgment happens, approved answers the only quotable layer, the index generated and disposable. ``` kb/ ├── raw/ <- the export. RESTRICTED. never leaves this folder ├── scrubbed/ <- redacted working copy. everything downstream reads this ├── curation/ <- candidates, conflicts, decisions ├── approved/ <- the only quotable layer ├── taxonomy/ <- categories, part names, customer wording ├── index/ <- generated. never hand-edited └── evals/ <- the test questions ``` ``` printf 'raw/\nscrubbed/\n*.pst\n*.mbox\n*.eml\n' > kb/.gitignore ``` **Why the ignore rules come first** — One build tracked an unredacted customer corpus in a private repo from the first commit and did not notice for months. Once it is in history it is in every clone and every backup. Two minutes now, or a bad week later. **Gate — all of these must be true before the next station:** - Your agent runs and can read the folder - raw/ and scrubbed/ are ignored before the export is copied in - The export is in raw/ and nowhere else **What you do here:** Create the structure. Write the ignore rules BEFORE anything is copied in, then test that they work. **What you must ask the human:** - Where does the export live now, and has it been copied anywhere else already? **What you must never do:** - Copy, move or read the export before the ignore rules are verified. - Assume an ignore file works because it exists. Test it. - Put the raw layer anywhere that syncs or backs up without being told to. ### Station 3 — Redact the export Deterministic, auditable, and it must prove its own coverage before you continue. Who does it: mechanical A model asked to redact will miss records quietly, and coverage cannot be proven afterwards — which is exactly the property you need here. A pattern-based scrubber is auditable: you can read every rule and say what it does and does not catch. ``` python scrub_pii.py --dry-run # counts only, writes nothing python scrub_pii.py # writes the redacted copy ``` **What pattern-based redaction does not catch** — Personal names in free text. “Thanks, Dave” survives — there is no reliable pattern for a first name. Bare city names survive too, deliberately, because masking them wrecks the text for counting. The scrubbed copy is reduced-risk, not anonymous. It stays internal. **Gate — all of these must be true before the next station:** - The leak report is clean for both emails and phone numbers - Two copies exist: an untouched archive, and the scrubbed working copy - Every file was handled — nothing silently skipped as an unrecognized type **What you do here:** Run the script and read its report back. You are the operator here, not the redactor. **What you must ask the human:** - Which email domains and phone numbers belong to the business, so they are preserved rather than masked? - What format is the export in? Mail-archive formats have to be converted first. **What you must never do:** - Redact by reading the text yourself. - Proceed if the leak report is not clean on every category. - Describe the scrubbed copy as anonymous. - Skip an unrecognized file type without reporting it loudly. Anything skipped is unredacted. ### Station 4 — Count what customers actually ask Extract verbatim, cluster by need, rank by volume times what a wrong answer costs. Who does it: mechanical + a second reader Do not start by writing answers. Start by finding which questions are worth answering — demand is heavily skewed, and at the reference build one category was over 40% of everything. Rank by two things multiplied: **how often it comes up**, and **what a wrong answer costs**. A rare question that scraps a part outranks a common one that does not. ``` Read every conversation in scrubbed/ and extract the questions customers actually asked, in their own wording. One row each: question | their wording, verbatim intent | what they were really trying to find out answer | what we told them, if we did source | file and record id Write to taxonomy/questions.jsonl. Do not summarize, interpret or merge yet. I want the raw distribution first. ``` ``` Read taxonomy/questions.jsonl and group into recurring issues. - Group by what the customer needed, not how they worded it - Keep every distinct phrasing as a variant of its cluster - Flag clusters where our past answers DISAGREE with each other - Flag clusters about parts or policies that no longer exist Sort by volume. Give me the disagreement flags separately. ``` **The disagreement list is the valuable output** — Clusters where your own past answers contradict each other are where the policy was never written down, or changed and nobody propagated it. That list is worth more than the volume ranking. **Gate — all of these must be true before the next station:** - A ranked list of 20–30 real intents, in customer wording - A second person has read it for what is missing - Clusters touching discontinued parts or superseded policy are marked, not deleted **What you do here:** Extract questions verbatim, one row each. Then cluster by what the customer needed. Produce a ranked list and a separate disagreement list. **What you must ask the human:** - Which of these clusters touch products or policies that no longer exist? **What you must never do:** - Summarize or merge on the first pass. Extract verbatim first, cluster second. - Group by how the business is organized. Group by what the customer needed. - Discard clusters where past answers contradict each other — surface them. - Rank by volume alone. Weight it by what a wrong answer costs. ### Station 5 — Write down what is true Roughly thirty statements, each with its exceptions. This is the one that cannot be delegated. Who does it: the owner — the constraint Write down the **roughly thirty statements that must never be wrong** — the return window, the standard lead time, the warranty terms, the tolerance you actually hold — and beside each, **its exceptions**. The exceptions are the whole job. At the reference build, the most load-bearing rule in the entire return policy — that nothing ships out before the original comes back and is inspected — appeared in no file and on no page anywhere, until the owner said it out loud in a single sentence. ``` statement: Returns are accepted within 30 days, and require authorization before anything ships back. exceptions: - International orders are final sale. - Restocking fee 10% unopened, 20% opened. - Nothing ships out ahead of the return coming back. - Customer-modified parts are not accepted. risk_class: policy approved: <-- stays empty until a person signs it ``` **Why this station cannot be handed to a model** — A governing statement cannot be validated by the process that consumes it. If a statement is wrong, every answer consistent with it gets cleared confidently and calibration will not detect the error. This is the one input no downstream check can catch, which is why the owner is permanently on the critical path. **Gate — all of these must be true before the next station:** - Roughly thirty statements written, each with its exceptions - Every statement carries a risk class assigned by its source - A person has signed them — an unsigned set is inert - Topics with no authority are on an explicit unresolved list, not quietly guessed **What you do here:** Interview the owner and draft candidates from their answers. Everything you draft stays marked draft with an empty approver until they sign. **What you must ask the human:** - For each high-volume cluster: what is the rule, and what are its exceptions? - When does the normal answer stop applying? - What would a new hire get wrong here? - Which of these has never actually been written down anywhere? **What you must never do:** - Write a governing statement from the history. History is evidence; the owner is the authority. - Fill a gap the owner left. If they do not answer, it goes on the unresolved list. - Drop an exception to make a statement read more cleanly. - Mark anything approved. You cannot approve. ### Station 6 — Check the history against it Four verdicts. Only one clears, and only outside the high-risk classes. Who does it: mechanical + calibration Now the volume gets processed. Each past answer is checked against the signed statements and sorted into exactly four verdicts. Judged against this signed statement: Fits bars 0.75″–2.1″ tall. Exception: 1.6″–2.1″ needs the included spacers. Over 2.1″ does not fit. - **supported** — “a 1.25″ bar works”. Agrees. No exception applies. → clears - **incomplete** — “a 1.8″ bar fits fine”. True — and drops the spacer exception. → held - **contradicted** — “2.5″ bars are fine”. Above the maximum. Disagrees outright. → held - **uncovered** — “aero bars are fine”. No signed source mentions aero bars. → abstain _incomplete_ is the dangerous one: the reply is true and still wrong, because it sends a customer away without the spacers they need. ``` supported agrees with a statement and its exceptions -> clears incomplete agrees but drops an exception -> hold, fix contradicted disagrees with a statement -> hold, review uncovered no citable source addresses this -> unresolved list ``` ``` Take 30 items the automatic check cleared. Have the owner grade each one blind. Compare. If you disagree on more than a handful, the statements are ambiguous -- fix the statements, not the checker -- and re-run. If they still disagree, the model is not good enough for this step. Try another. That is the whole model-selection process: measure it on your own material. ``` **Calibrate on entailment, not on taste** — One build shelved a working checker because it scored 0.55 against an arbitrary 0.7 threshold — on a sample of ten, grading tone rather than whether a claim followed from a rule. Ten items cannot demonstrate anything either way, and tone is not what makes an answer wrong. **Gate — all of these must be true before the next station:** - A calibration sample the owner graded, agreeing closely enough to trust the rest - Every high-risk verdict routed to a person regardless of the checker's output - The uncovered list handed back to the owner as new statements to write **What you do here:** Assign exactly one of the four verdicts to each historical answer. Report the distribution. **What you must ask the human:** - Grade this calibration sample blind, then tell me where we disagreed. **What you must never do:** - Clear anything in policy, pricing, warranty or safety, whatever the verdict says. - Invent a fifth verdict, or soften contradicted into partially supported. - Scale up before the calibration sample agrees. - Report agreement rates you have not measured. ### Station 7 — Build the answers and the index One article per reusable issue. The index is generated and never hand-edited. Who does it: mechanical One article per reusable issue — not one per email. The article is the answer that gets reused; the original conversation stays behind it as evidence. The index is generated from approved articles and disposable. When an answer changes you edit the article and rebuild. ``` # The customer issue ## Applies when ## The answer ## What to collect first ## Exceptions and limits ## Escalate when ## Sources ``` **Never hand-edit the index** — One build corrected a discount rate directly inside the generated index. The source kept the old number, the drift check compared file hashes and saw nothing wrong, and it reported clean for 73 days while the wrong figure went to customers. **Gate — all of these must be true before the next station:** - Every indexed article is approved and cites its source - The index rebuilds from scratch, reproducibly - Nothing archived, expired or unapproved is retrievable **What you do here:** Write one article per reusable issue, from approved statements only. Generate the index. Report counts against the previous build. **What you must ask the human:** - Which candidate articles is the approver signing, and which are waiting? **What you must never do:** - Write an article for an issue with no approved statement behind it. - Edit the generated index for any reason, including to fix something you know is wrong. - Include archived, expired or unapproved material. - Write a reduced index and call it complete. ### Station 8 — Draft, review, send Prove it offline against real questions first. Then a person reads every reply. Who does it: normal support work The assistant pulls the thread, the approved answer, that customer’s own order record, and the rules about what it may say — then writes a draft with its sources attached. A person reads it, edits if needed, and sends. Before any of that, prove it offline. ``` 50-100 questions, taken from Station 4's actual customer wording. Include, deliberately: - ordinary phrasing, and typos - exact part and model numbers - ambiguous questions - questions with NO supported answer <- the important ones - advice that has since been superseded - anything restricted or internal-only ``` **The habit that decides whether this lasts** — When an article turns out wrong or thin, fix it or flag it right then, while the ticket is open and you still remember why. That one habit is the difference between a library that stays alive and one that quietly rots into another thing nobody trusts. **Gate — all of these must be true before the next station:** - A 50–100 question test set built from real demand, not invented - Abstention verified on unsupported questions, especially high-risk ones - A named person reviews every reply during the pilot - A review cadence and an owner for keeping articles current **What you do here:** Draft from approved articles, attach citations, hand to a person. Test against the real question set before any of this goes live. **What you must ask the human:** - Who is reviewing every draft during the pilot, and have they agreed to it? **What you must never do:** - Send anything. - Commit to an outcome — a refund, a return, a warranty decision, a delivery date, an expedite. - Answer from memory of the conversation rather than a retrieved, cited article. - Guess when the articles do not cover it. ## Why the gates exist Every rule above came from something breaking. These are the failures that produced them. ### The check was working. It was checking the wrong thing. A generated index entry was hand-corrected to 5% while its source document kept the old 10%. Both sides hashed the same untouched file, so the drift check matched and reported clean every day for 73 days. Hashing a file tells you the file moved. It cannot tell you a number inside it changed, and it cannot see that a generated entry now contradicts the document it came from. **The rule this produced (Station 7):** Fix the source and rebuild, never edit the generated layer. If a tool lets you edit it at all, that is a defect in the tool. ### Fixed is not live. A correction moved through the authority file, the FAQ source and the publish script — all three correct — and stopped, because nobody ran the publish step. The only surface a customer could see stayed wrong for 4 days. Worse: three entries on that live page existed in no internal document at all, so the public page had quietly become a source of truth nobody was maintaining. **The rule this produced (Station 8):** The publish step is part of the gate, and you verify it landed rather than assuming. ### A build lost a third of the index behind a green checkmark The index builder needed a credential to fetch part of its content. Without it, the builder dropped a whole tier, exited successfully, and every automated check passed. The output was 179 entries instead of 262 — not obviously broken, just a different plausible-looking shape. The one guard that should have caught it was a count range that did not contain the correct value either, so it warned on every good build and got ignored as noise. A check that cries wolf on correct runs cannot catch the broken one. **The rule this produced (Station 7):** A partial result is a failure, not a smaller success. Compare counts against the last run and stop rather than shipping a reduced output. ### Machine summaries invert details A summary in the evidence layer said the _rear_ mounting points were the hard ones to reach. The owner corrected it — the _front_ ones are, and that distinction was the single largest driver of returns on the product. The records were machine-generated summaries, so the inversion was an artifact rather than a real historical answer. Every other directional or dimensional claim in the same corpus carries the same class of error. ### Corroboration is not authority Fifty past emails saying the same thing tells you what was said fifty times. It does not tell you it was correct, and it does not make it policy. Repetition can promote an observation about how something behaves in the field. Only someone with authority to set the position can establish the position. ### Silence blocks — it does not default A discount code appeared repeatedly in historical records and not one of them stated its percentage. The temptation is to infer one from context. A guessed number is indistinguishable from a real one once it is in the index, and it gets quoted with total confidence. Two different rates were live at the same time before this was caught. ### Your live site may be more current than your internal files An audit concluded three policies were missing — exchanges, damage on arrival, and a returns process. All three were published and live. The internal documents were the stale copy. Read what your customers can actually see before concluding a policy does not exist. Three out of three of those findings reversed on it. **The two rules they all reduce to.** A repeated historical answer is not policy — corroboration establishes what was _said_, never what is true. And silence blocks rather than defaults: where nobody wrote a number down, the answer is unresolved, not a sensible-looking guess. ## What the assistant is never allowed to decide - **Approve a return or a refund** → instead: State the policy and the window, list what the customer must provide, route it - **Issue an authorization number** → instead: Explain that authorization is required and how it is requested - **Decide a warranty or quality claim** → instead: State the coverage, the exclusions, and what evidence a claim needs - **Commit to a date or an expedite** → instead: State the standard lead time, then route to whoever can commit - **Quote a current price or quantity** → instead: Read it live from the system of record, or say it cannot and route - **Send anything by itself** → instead: Draft, cite its sources, and wait for a person **The general rule.** An assistant may describe what is already written down, collect what is missing, and route. It may never commit the business to an outcome. If a sentence would oblige you to do something, a person writes it. ## Honest limits - Not free. A paid assistant subscription covers the conversational stations; the bulk passes over your archive are API credits, billed per token, and that is where the spend concentrates. - Not local. Classification and drafting are model calls, and partially-redacted customer text leaves the building to reach the model. - Not anonymous. Pattern-based redaction catches structured identifiers, not a customer name typed mid-sentence. - Not enforced. The gates are conventions. “Approved” is a word somebody types; nothing blocks a build or refuses a publish. - Not a phase you graduate from. Policy, pricing, warranty and safety stay under human review permanently. - Not a live-system tool. Order status and stock need the system that owns them. - Not a substitute for deciding. It will not fix a business that never wrote its policies down. It will show you that clearly, which is not the same thing. The reference build is Stark Side Gear (https://tentmount.com) — its figures and its defects are real and quoted as measured. Figures verified 2026-08-11; the build is still in progress, with roughly thirty governing statements drafted and 0 approved. ## About these figures Volumes on this page are deliberately given as ranges. Only the figures that carry the argument are exact: - the 10% versus 5% discount discrepancy — without both numbers there is no discrepancy - 73 days of a hash check reporting clean while the wrong number went to customers - 4 days the live page stayed wrong after every internal file was fixed — one error is an anecdote, two is a mechanism - 262 chunks on a good build against 179 on a degraded one that still exited successfully - 0 approved governing statements, which is the honest number and is never rounded up This is the method applied to itself. A precise public figure carries two permanent obligations — a defensible counting method, and somebody to re-derive it forever — and a volumetric earns neither. Two figures here were wrong within a day of first publication: one was a count that reproduced from no definition at all, and one was a total that had already moved. Both were the kind this page now states as ranges. If you are reproducing this method, apply the same rule to your own write-up.