How users try to break your AI assistant, and what protects you.

The testing starts the day you go live

Anyone who has run an AI advisor in customer contact knows the rhythm. The first conversations are what you hoped for: product questions, sizing questions, availability. Then, usually within days, a different kind of message shows up. Someone asks the assistant to contradict itself. Someone claims a discount that was never offered. Someone tries to talk it out of its role. Most of this is plain curiosity. People poke at a new thing to see where it bends. A smaller part is opportunism: if the assistant can be maneuvered into saying something useful, a screenshot of that answer might be worth money. Professionals have noticed the opportunity too. According to signals identified by Accenture, 78% of financial institutions expect an increase in fraud connected to AI agents, that is, AI systems acting on a user's behalf.

One thing before the examples: they are deliberately harmless. They show what users attempt, seen from the outside, and none of them is a working recipe. This article is not a manual for breaking assistants. The goal is that you recognize the categories, because every one of them has a structural answer, meaning an answer built into how the system works rather than into how the model behaves.

The five patterns you will see

Two of these deserve a closer look. Discount extortion is the most common and the most expensive. It does not need the assistant to be wrong, only polite. Chat models are trained to be agreeable, so when a user smuggles a false claim into the question ("you just promised me 50% off"), the model often confirms it just to keep the conversation friendly. The screenshot of that confirmation looks exactly like a real commitment. False-statement bait runs the same trick in the other direction: the user states something flattering but untrue about your product and waits for the assistant to nod along. If it agrees that your product is suitable for children, or allergy-safe, or certified, and your own material never said so, you have a liability problem in the shape of a chat bubble.

The fifth pattern needs one term explained first. A hallucination is what happens when an AI has no real answer and fills the gap with invented details that sound plausible. Every assistant does this occasionally on its own; hallucination fishing means users trigger it on purpose. Ask detailed questions about a product line that never existed, and a poorly protected assistant will describe it in confident detail, quotable later. Why this happens and what stops it is covered in how to stop an AI assistant from making things up.

Why a stricter system prompt will not save you

The reflex answer to all five patterns is a stricter system prompt. Quick definition: the system prompt is the hidden instruction text the model reads before every conversation, things like "never promise discounts", "never reveal internal data", "stay in your role". Writing those rules helps, and you should do it. But an instruction is not a wall. The model tries to follow it, and a determined user tries to talk the model out of it, with unlimited attempts, at zero cost, on your infrastructure. Sooner or later one attempt lands. The protections that hold are the ones that keep working even when the model has a weak moment.

Notice what these five layers have in common: none of them asks the model to resist temptation. They remove the temptation from the model's reach. An assistant that cannot see your purchase prices cannot leak them to a fake managing director. An assistant whose every publishable answer passed a human review cannot "confirm" a 50% discount, because no reviewed answer contains one.

What this looks like in practice

This checklist is tool-neutral. You can hold it against any product you evaluate. In Klariton it maps to specific places. The Chat Advisor answers only from reviewed material, structured as question, answer and source, a format we call a BIQ. Every answer also passes a review gate, meaning a person approves it before it can ever reach a visitor. The Compliance Center keeps the operational record, including logged incidents when someone tries one of the patterns above. And SafeGuard, the checking layer, re-checks published answers against your rules and sources on an ongoing basis, so nothing drifts out of bounds after release.

Whichever tool you choose, ask the vendor the blunt versions of these questions: what happens when a user claims the assistant promised a discount, where do I see the attempt, and what stops the assistant from stating a price that is not in my data? Evasive answers are an answer.

Frequently asked questions

Is a system prompt enough protection against manipulation?

No. A system prompt is the hidden instruction text the model reads before every conversation, and users who test your assistant treat instructions as a challenge to get around. A good prompt makes manipulation rarer, but it guarantees nothing, and you cannot audit it after the fact. Protection that holds is built into the setup instead: the assistant only draws on material you reviewed, cannot state prices or conditions that are not in your source data, shows a visible note that its answers are not binding, logs abuse attempts, and has its published answers checked against your rules on an ongoing basis.

Are you liable for what a chatbot promises a customer?

That depends on the jurisdiction and the details of the case, and courts have already dealt with disputes over chatbot statements. This article is not legal advice; for your own situation, talk to a lawyer. What you can do on the operational side is shrink the surface. Tie answers to approved source material, so the assistant cannot state conditions that were never released. Add a visible note that answers are informational and that binding terms come from the order process or a human. And keep records of what was said, so a screenshot never stands against an empty file. The cheapest dispute is the one your setup prevented.

How do I know whether users are already testing my assistant?

Only through logs. People who test limits do not announce it, so without conversation records and incident logging you will not see it happening. That makes logging a protection layer of its own, not a nice-to-have: it shows you which patterns your visitors actually try, gives you the full exchange if a screenshot ever surfaces, and points you to where your source material and rules need tightening.