How to use the Knowledge Optimizer completely
Diagnose a weak answer, improve the right layer and prove the result with the same test.
A weak answer can originate from missing knowledge, poor retrieval, the role prompt or the selected model. The Knowledge Optimizer separates those layers so you can fix the cause instead of guessing.
This verified workflow first records a successful PDF baseline, then tests a deliberately unsupported warranty case and keeps only changes that improve the same fixed questions.
What you will have at the end
- A documented baseline failure
- A targeted content, role or model improvement
- A verified before/after result and a clean inbox
Before you start
- An indexed knowledge base
- One weak or unanswered real question
- Enough message quota for repeated tests
Optimize the failing layer
Knowledge Test shows retrieved evidence, Optimize improves source material, Role refines behavior, Model Arena isolates model differences and Audit checks broader quality. Inbox keeps suggested improvements reviewable.
Changing provider in Model Arena clears incompatible embeddings and requires re-indexing; account for that before a comparison.
01–05
Set it up step by step
Record a complete successful baseline
Run five fixed PDF questions and keep every result visible.
In Knowledge Test, run the five NORDSTERN-42 paraphrases before changing content, role or model. The captured result keeps the 5/5 score and all five grounded answers visible in one frame. This gives the later negative warranty test a known-good retrieval baseline.
Understand the Optimize inbox
Real conversation gaps appear here; controlled tests do not fabricate traffic.
Open Optimize to review suggestions derived from real visitor conversations. The clean tutorial account intentionally shows an empty inbox because only controlled Knowledge Tests were run. In production, verify the original conversation and authoritative source before accepting a generated content change.
Generate and review a role proposal
Compare current role, proposed role and rationale before testing.
Use the failed warranty variants to generate a proposed role. The captured result shows the current and proposed instructions side by side with the optimizer rationale. Do not apply it yet: a plausible proposal is only a hypothesis until the same questions pass an A/B test.
Compare measured Model Arena results
Read quality and median latency from the same five questions.
The real Model Arena run reused the five PDF verification questions for Vertex Gemini 2.5 Flash and Claude Haiku 4.5. Both scored 100%; Gemini had 2.6 s median latency versus 4.4 s for Claude, so the current model was recommended. Inspect individual answers before accepting the aggregate score.
Reject a worse role after the A/B test
A generated change is not automatically an improvement.
Run the exact five questions against the current and proposed role. The captured A/B result shows 0% better, 3 ties and 2 worse. Do not apply this proposal; keep the current role and add the missing warranty information to an authoritative source if the business actually offers such a policy.
Example & result
See the practical test and its result
Every tutorial includes a fixed input, the expected outcome and a transparent record of what was actually verified locally.
Practical example: How to use the Knowledge Optimizer completely
This exact scenario was completed with the temporary tutorial account.
Exact test input
What is the unique verification code in the uploaded tutorial PDF?
Expected result
All variants are answered and contain NORDSTERN-42.
What was actually verified
The real Knowledge Test scored 5/5 answered (100%); every answer contained NORDSTERN-42.
Tips & tricks
Make the setup reliable
Test with realistic examples, record your baseline and change one setting at a time. That makes real improvements visible.
Keep a regression set
An improvement for one question must not break other important questions. Re-run a small fixed set before publishing.
Never auto-accept factual rewrites
AI suggestions can simplify away exceptions or dates. Compare every factual change with the authoritative source.
When something does not work
Troubleshooting
Check status, permissions and test data systematically before changing the model or prompt.
The expected option is missing
Confirm the account plan, feature permissions and selected chatbot. Paid or beta features can be hidden when prerequisites are not met.
The test result is inconsistent
Reset the test conversation, keep the input identical and change one setting at a time so the cause remains measurable.
Ready for a production-style test
Add the verified question to a recurring regression set and review the Optimization Inbox on a fixed schedule instead of making ad-hoc changes.