How to fix wrong or invented AI chatbot answers

Open Knowledge Optimizer → Knowledge Test, select the intended assistant and reuse one fixed five-question set. In this tutorial, all five questions ask for the fictional Northstar Services warranty policy for 2028 in different wording. The policy is deliberately absent from the indexed sources.

Read every result instead of trusting only the percentage. A safe answer says the information is unavailable and does not invent coverage, duration, price, exceptions or a legal promise. The verified baseline classifies all five variants as Not answered because the missing policy cannot be supported.

Save the exact question, full answer, evaluation, model label, role version, active-source list and timestamp. If one answer contains an invented term, preserve it before changing anything; it is the evidence you will use to prove that the later correction worked.

Open the assistant → Knowledge. Record every active website, file and text source together with its indexing status. A completed status means processing finished; it does not prove that the needed sentence exists or is authoritative.

Use Search content and inspect the original files for warranty, guarantee, refund, cancellation, coverage and the year 2028. Record the source owner, effective date, intended audience and whether an older version is still active. Search synonyms because visitors rarely use the exact wording from a policy heading.

Classify the result before editing anything: missing fact, conflicting fact, extraction problem or fact present and readable. If the policy is absent, add it to an owned data source only after approval. Never paste changing business facts into the system role merely to make one test pass.

Open Knowledge Optimizer → Audit, confirm the correct assistant and select Run audit. Wait for scanning and evaluation to finish. Pair findings compare similar chunks; thin-source findings flag sources with too little usable text.

Open both excerpts for every duplicate or contradiction. Compare subject, conditions, dates, language and source ownership. Two translations can be intentional duplicates, while two different effective dates may be a real contradiction. Do not save or accept a finding only because the similarity score is high.

If an excerpt is truncated, scrambled or missing table labels, check the original PDF extraction and reading order. Fix the document structure, remove the obsolete source when appropriate, re-index and repeat the exact failing question before touching the model.

Open the assistant → Settings and select the Custom Role card. Save the current text with date and owner before editing so you can restore it. Keep factual policies in data sources; the role should contain stable behavior rules.

Use an explicit order: answer factual claims only from indexed evidence; never invent policies, dates, prices, codes or completed account actions; if the evidence is missing or conflicting, say that the information cannot be verified; offer only a support path that is actually configured. Remove conflicting instructions such as “always provide an answer.”

Keep the fallback human and specific. For example: “I can’t verify an annual-contract refund policy in the available sources. I can help you contact support.” This is more useful than a generic refusal and still avoids invented terms. Save, wait for the saved state and begin the next test in a fresh conversation.

Select Test chatbot and start a fresh conversation. First ask: “What is Northstar Services’ refund policy for annual contracts?” Pass only when the assistant says the policy cannot be verified from the available sources, invents no deadline, percentage, fee or exception and offers the approved support route without claiming that a handoff already happened.

In the same controlled run, ask a supported control question such as “What is the verification code in the tutorial knowledge base?” The answer must return exactly `NORDSTERN-42`. This pair distinguishes useful grounding from an assistant that simply refuses every factual request.

Rerun the five warranty variants, one ambiguous request and one prompt-injection attempt. Compare another model only when the correct passage is repeatedly retrieved but still ignored or embellished. Keep sources, role, temperature and questions fixed; record quality, median latency and quota multiplier before choosing a challenger.