AI chatbot Knowledge Optimizer: complete tutorial

Open Knowledge Optimizer → Knowledge Test, choose the intended assistant and select Custom Question. Add five natural paraphrases that should all return the unique PDF fact `NORDSTERN-42`. Include the direct question, a polite variation and wording a real visitor might use.

Write down the expected fact, approved source and pass condition before selecting Start test. A passing answer must contain the exact code, identify the Northstar Services demo PDF and avoid invented warranty, price or availability claims.

Keep the same source version, role, model and temperature. Add a separate unsupported 2028 warranty question to verify safe no-answer behavior. This prevents a fluent answer or a changed test setup from looking like an improvement when it is not.

The recorded run shows 5 answered, 0 partial, 0 unanswered and a 100% score. Open every result and confirm that each answer contains `NORDSTERN-42`, names the Northstar Services demo PDF and adds no invented warranty, price or availability claims.

Evaluate two layers separately. First ask whether the correct passage was retrieved. Then judge whether the answer is accurate, supported by that passage, complete enough for the question and free of unsupported additions. A single overall score can hide a fluent answer that used weak evidence.

This proves that the indexed PDF and retrieval path work for the known fact. It does not prove that the chatbot knows a 2028 warranty policy. Ask that unsupported question separately and expect a safe “no information” response rather than a guess.

Open Optimize to review knowledge gaps derived from real visitor conversations. The verified tutorial account is empty because controlled Knowledge Tests do not fabricate customer traffic. In production, choose 30 or 90 days, run Analyze, open the original conversation and verify every suggestion against the authoritative owner and source before accepting it.

Use Audit when you need a wider review rather than one known failure. It scans for stale facts, contradictions, thin coverage and weak structure. Treat every finding as a prioritized checklist item: read the affected excerpt, confirm the latest approved fact and decide whether to edit the source or dismiss the finding. Never use Save to Inbox as automatic approval.

Empty Optimize results mean “no conversation-derived suggestion in this range”, not “the knowledge base is perfect”. The published real-click video runs Audit to completion and shows the genuine result. It also verifies that no finding was saved, no role was applied and no model was switched automatically.

Open Role and use the unsupported 2028 warranty variants to generate a proposal. The visible suggestion keeps answers short, requires indexed evidence, forbids extrapolation and tells the assistant to say when information is missing. Those are sensible behavioral guardrails.

Still, a role cannot create a warranty policy that does not exist in an approved source. Read Current role, Proposed role and Rationale side by side. Remove any instruction that conflicts with your support process, escalation path, required tone or legal wording.

Leave the proposal unapplied. Select the saved five-question test as the A/B source and compare current and proposed behavior on identical inputs. This isolates the role change; editing the source or model at the same time would make the result impossible to attribute.

Reuse the last Knowledge Test so both candidates receive the same five PDF questions. The recorded comparison shows Vertex Gemini 2.5 Flash and Claude Haiku 4.5 at 100% quality; median latency is 2.6 seconds for Gemini and 4.4 seconds for Claude, so the current model is recommended for this test.

Open the per-question answers before deciding. Check exact facts, evidence use, safe refusal, completeness and any unusually slow response. Compare quota or model cost as well as median latency; one fast median should not hide repeated slow outliers or weaker safety behavior.

A green recommendation is not a universal winner. Expand the regression set with long answers, ambiguity, refusal, language and tool-use cases that matter to your chatbot. Changing AI providers can require new embeddings and a full re-index, so back up the current setup and schedule that comparison separately.

Run the exact five warranty questions against the current and proposed role. The recorded result is 0% better, 3 ties and 2 worse. Open the losing pairs to understand whether the new wording became less accurate, less useful or less safely grounded. Leave Apply role untouched and keep the current role.

If the company truly offers a 2028 warranty, add the approved policy—with owner, effective date, scope and exceptions—to an authoritative source, re-index and repeat the known-answer and safe-no-answer tests. If no policy exists, the correct improvement is a consistent safe fallback or human handoff, not more confident wording.

Before publishing any change, record the accepted source version, role, model, five-question result, unsupported-question result and latency. Review the Optimization Inbox on a schedule and keep suggestions pending until a human owner confirms both the fact and the measured regression result.