Enterprise software · Customer support assistant
Giving a support assistant real authority, with evidence it won't be misused.
A global B2B software company wanted its support assistant to do more than answer questions: look up accounts, change seats and issue service credits. The question was whether it could be trusted with those tools in front of every customer.
Sector
Enterprise software (B2B SaaS)
System
Retrieval-backed support assistant with ticketing, account and billing tools
Engagement
AI Red Team and AI Security Assessment
Timeline
Six-week engagement; continuous assurance since
The situation
The next release gave it hands.
The support team had spent a year on a retrieval-backed assistant that answered from the help centre and policy documents. It worked, and customers used it. The next release connected it to live systems: it could read an account's plan and history, change seats, and apply service credits up to a set limit. Product was ready to ship. Security was not ready to sign.
Nobody disagreed about the value. The gap was evidence. The team could show that the assistant answered well; it could not show what happened when a customer, or a message a customer pasted in, tried to steer it. They asked us to find out before their customers did.
What we found
The conversation could reach the tools.
Ticket content steered the assistant's tools
Text inside a customer's own ticket - a pasted email, a forwarded message - could instruct the assistant to apply a credit to a different account the customer named. The assistant treated the instruction as part of the conversation. Four distinct paths reached the billing tool this way.
Account boundaries lived in the prompt, not the system
The assistant was told which account it was serving; it was not technically restricted to it. Under pressure it looked up and summarised another organisation's plan details.
Retrieval surfaced internal-only guidance
Help-centre search also indexed internal escalation playbooks. The right phrasing returned them verbatim, including the conditions under which staff are allowed to waive fees.
The credit limit was a suggestion
The per-request cap was enforced in the prompt. Splitting a request across several turns passed it.
What changed
Authority moved out of the prompt and into the system.
Every validated path was closed at the layer that could actually enforce it, and the assistant shipped with tool access in the next release - with evidence the security team could sign.
- Account scope enforced at the API layer, derived from the authenticated session rather than the conversation
- Billing and account tools placed behind a policy service with hard limits and human approval above a threshold
- Customer-submitted content tagged and isolated from instruction context before it reaches the model
- Internal playbooks removed from the customer-facing retrieval index
- A reproducible attack suite covering every validated path, handed to the engineering team
Ongoing assurance
Every release is tested the way an attacker would.
The assistant now ships every few weeks. Each release, model update and prompt change runs through the attack suite and a behavioural regression set we maintain together. When a model upgrade quietly changed how the assistant handled pasted content earlier this year, the regression caught it before release.
We review results with the team each quarter and extend the suite as new tools are added.
How continuous assurance worksMore case studies
All case studiesBegin with evidence