How Does AI Customer Service Work at Enterprise Scale?

At enterprise scale the hard part is not the conversation — it is grounding every answer in your actual policy and systems, and proving afterwards what was said and why. An enterprise-grade deployment answers from your documented sources, respects the entitlements of the person asking, and hands anything outside its remit to a specialist with the full context attached. In the EU it also has to tell the customer they are talking to an AI. Article 50(1) of the AI Act puts that duty on the system's provider — and an assistant put into service under your own name makes you the provider, which is the normal case for a bespoke build. So the disclosure is a design requirement rather than a disclaimer.

Consumer Chatbot vs. Enterprise Deployment

Ground It, Then Cite It

A model answering from memory will eventually state your policy incorrectly with total confidence. At consumer scale that is embarrassing; on an enterprise contract it can be a breach.

The fix is retrieval: the system finds the relevant passage in your live documentation, answers from it, and shows which document it used. Your agent or your customer can check it in seconds, and a wrong answer becomes traceable rather than mysterious.

It also solves the freshness problem. When a policy changes you update the document, not the model, and the next answer is correct.

Entitlements Are Not Optional

Different customers have bought different things. An answer that is correct for a premium contract can be wrong or actively harmful for a standard one, and a system that does not know the difference will confidently promise something you do not owe.

So the retrieval layer has to be permission-aware from the start. What the model can see for a given conversation should be scoped to that customer's entitlements — retro-fitting this after launch usually means rebuilding the retrieval design entirely.

Disclosure and Erasure: EU AI Act Article 50(1) and GDPR Article 17

Article 50(1) of the EU Artificial Intelligence Act, Regulation (EU) 2024/1689, opens “Providers shall ensure that AI systems intended to interact directly with natural persons are designed and developed in such a way that” those persons are informed they are dealing with an AI system, unless that is obvious to a reasonably well-informed and observant person in the circumstances. Note the subject. The duty sits on the provider, not on the deployer — Article 50's deployer duties are 50(3), emotion recognition and biometric categorisation, and 50(4), deep fakes and public-interest text. Neither is a chatbot disclosure. That distinction rarely gets an enterprise off the hook, because Article 3(3) defines the provider as whoever places a system on the market or puts it into service under its own name or trademark: an assistant that ships as your brand makes you the provider of it. So design the disclosure in rather than contract for it. At enterprise scale the risk is then the hand-off. A session that starts with an AI, escalates to a human and comes back has to re-establish the disclosure. Model it as a property of the conversation participant and render it in the transport, so it cannot be lost in a routing change or overridden by a persona prompt.

Article 17 of the GDPR, Regulation (EU) 2016/679 — erasure without undue delay — is the requirement grounded, retrieval-based support architectures most often fail, because the personal data is not only in the transcript store. It is in the vector index built from transcripts, in the fine-tuning set, in the evaluation corpus and in the analytics warehouse. A deletion that reaches only the primary store leaves the content retrievable, which is the same failure in substance. So the constraint is a deletion path that fans out to every derived store, with enough derivation lineage recorded to compute the fan-out. Note that 17(1)(c) makes an Article 21(2) objection a deletion trigger, not only a stop-processing one.

Which Models We Would Shortlist for This

Enterprise support is a tiered system, not one model. Most volume is classification and routing; a small remainder is regulated conversation where the transcript is subject to the same controls as the records it discusses.

Claude Sonnet 5 — $3/$15 flat across 1,000,000 tokens with a published 1.1x multiplier for US data residency. The escalation tier for regulated accounts, where conversation transcripts carry the same obligations as the underlying records.

Claude Haiku 4.5 — $1/$5 with a 200,000-token window and a 50% batch rate. Anthropic publishes the 1.1x US data-residency multiplier on Claude 4.6 and later, and we have not found it published for Haiku 4.5 — so where residency is a hard requirement, keep the regulated escalation on Sonnet 5 unless Anthropic confirms otherwise.

Gemini 2.5 Flash-Lite — $0.10/$0.40 for the classification and routing layer in front of the rest of the stack, where most of the request volume is absorbed.

Mistral Large 3 — Apache 2.0 at $0.50/$1.50 hosted, and self-hostable where customer conversation data cannot be processed by a US provider at all.

Prices are the providers' own published list rates, not resale or routed rates. Each model page names the source document and the UTC time the figure was checked. Claude Sonnet 5 is quoted at its standard rate; Anthropic's $2/$10 introductory rate runs to 2026-08-31.

Where This Fits

This is one part of our work in AI for Customer Service. See the full set of AI use cases for the equivalent in other industries and functions.

Frequently Asked Questions