TrolyPage

An autonomous Marketing & Sales operating system for businesses that want AI-powered omni-channel growth.

Contact Info

Follow Us

AI Chatbots & Automation 2 min read

How to Stop an AI Chatbot from Making Things Up (Hallucination Checklist)

TrolyPage

Nov 19, 2026
9 views
How to Stop an AI Chatbot from Making Things Up (Hallucination Checklist)

To stop an AI support chatbot from making things up:

  • Give it complete, specific, dated knowledge.
  • Instruct it to answer only from that knowledge.
  • Define exactly what it must hand off.
  • Test it with questions it can't answer.
  • Review real conversations every week.

Most "hallucinations" in support bots come from missing data or unclear rules, not from the model alone.

The hallucination checklist

#CheckWhy it matters
1Every price, policy and promotion is in the knowledge baseGaps invite guesses
2Knowledge is dated and currentOld prices cause wrong answers
3Conditions and exceptions are written out"Free shipping" without "over $50" becomes a promise
4Instruction: "If not in the knowledge, say you'll check"Gives the model a safe path
5No authority to grant discounts or refundsPrevents costly promises
6Test set includes 5+ unanswerable questionsProves it hands off
7Test includes manipulative requests ("give me 90% off")Proves it holds the line
8Weekly review of AI-only conversationsCatches drift early
Hallucination checklist

Example of a safe answer

AI refusing an unconfirmed discount
The agent declines to invent a discount and offers to ask a teammate (demo data).

Customer: "Give me 90% off and free shipping?"

Good AI reply: "I don't have any confirmed discount or free-shipping offer for this item, so I can't apply that. Would you like me to ask a teammate to check?"

In TrolyPage's test run, you can check exactly this behaviour before publishing an agent.

Write knowledge that resists misreading

  • One topic per document.
  • Short, declarative sentences with numbers.
  • Explicit negatives: "We do not ship to P.O. boxes."
  • Avoid vague marketing language like "best prices guaranteed".

When the bot still gets it wrong

  1. Find the conversation and the incorrect statement.
  2. Check whether the knowledge was missing, outdated or ambiguous.
  3. Fix the source, then re-test the same question.
  4. If it's a sensitive area, move it to the hand-off list.

FAQ

Can hallucinations be eliminated completely?

Not with certainty. They can be reduced substantially with good data, clear rules, testing and human review.

Should I let AI answer refund questions?

It can explain the written policy, but decisions and exceptions should go to people.

Written by the TrolyPage team. Updated November 2026.