AI agent governance: guardrails for agents in real workflows
How to put AI agents into real workflows with enforceable limits: autonomy levels, tool permissions, human confirmation, audit logs, budgets, a kill switch and Arabic testing.
AI agent governance is the set of limits that decides what an agent may do on its own. It covers which tools the agent can call and with what permissions, where a person must confirm, how every action is logged, how much it may spend and how it is stopped. Write those limits into code and process before go-live, test them in Arabic and English, and start each workflow at the lowest level of autonomy that still saves real time. This guide covers autonomy levels, permissions, confirmation points, audit logs, budgets, the kill switch, prompt injection and customer-facing chat and voice agents.
What changes when AI moves from answering to acting?
A chat assistant returns text that a person reads and then acts on. An agent calls tools itself: it reads a CRM record, issues a refund, books an appointment or sends a message. The risk moves from a wrong answer to a wrong action, and a wrong action can reach customers, money and records before anyone notices.
The OWASP Top 10 for LLM Applications 2026, published in August 2026, ranks Prompt Injection first and Excessive Agency third. OWASP defines Excessive Agency as the weakness that lets an LLM-based system take damaging actions in response to unexpected, ambiguous or manipulated outputs. It traces the problem to three root causes: excessive functionality, excessive permissions and excessive autonomy.
OWASP's separate Top 10 for Agentic Applications, released in December 2025, adds risks specific to agents, including Agent Goal Hijack, Tool Misuse and Exploitation, Identity and Privilege Abuse, and Rogue Agents. Each of these maps to a control you can design: fewer tools, narrower permissions and clear points where a human decides.
Set autonomy levels before you choose tools
Agree the level of autonomy per action, not per agent. The same agent can draft freely, update a ticket on its own and still need approval to move money. This guide uses five levels.
| Level | What the agent does | Example | Typical fit |
|---|---|---|---|
| L0 Suggest | Drafts or recommends; a person does the rest | Drafts a reply for a support agent to edit | New workflows and regulated decisions |
| L1 Act with approval | Prepares the action; a person confirms each one | Prepares a refund that a supervisor approves | Payments, external messages, record changes |
| L2 Act within limits | Acts alone on low-impact, reversible actions under set thresholds; escalates the rest | Reschedules an appointment; escalates cancellations that carry a fee | Proven workflows with clear thresholds |
| L3 Act and report | Acts within a defined scope; people review samples and exceptions | Classifies and routes inbound email; a team lead reviews a daily sample | Mature, well-tested internal workflows |
| L4 Autonomous | Acts with no routine human review | Tidies duplicate tickets in an internal queue | Low-risk internal tasks only |
OWASP describes the same idea as graduated enforcement: low-consequence or easily reversible actions can be approved automatically, while irreversible ones go to human review. Its example is a customer service chatbot that issues a refund as store credit on its own but routes an external payout to a person.
Tool permissions, confirmations, audit logs and a kill switch
Permissions
- Give the agent only the tools the workflow needs, and remove tools left over from testing.
- Prefer narrow tools to open-ended ones. A tool that writes one type of file is safer than shell access, and a tool that summarises email should not be able to send it.
- Connect each tool with a least-privilege identity: read-only where possible, scoped to the tables, folders or mailboxes involved.
- Run actions in the context of the user the agent is serving, so it cannot reach another user's records.
- Enforce authorisation in code or in the target system. The model should never be the component that decides whether an action is allowed.
Human confirmation points
Require a person to confirm payments and refunds above a threshold, any message sent outside the organisation, deletions, changes to customer or employee records, and anything you cannot undo. Show the confirmation in the user's language with the exact action and values, such as the amount, the account and the recipient, rather than a summary.
Audit logs
Log every step so you can reconstruct what happened. Record the timestamp, agent and model version, user or session and the sources the agent read. Then record each tool call with its parameters and result, who approved what, and the cost of the run. Mask personal data that an investigation will not need, and set a retention period that matches your records policy.
Budgets, rate limits and a kill switch
Agents run in loops, and a confused or manipulated agent can call a model or a tool many times in a few minutes. OWASP lists Unbounded Consumption as LLM06:2026, including denial of wallet, where an attacker drives up pay-per-use costs. It calls for token-aware cost controls, hard spending caps and circuit breakers at the agent level.
- Per-run caps: a maximum number of steps, tokens and tool calls for a single task.
- Daily budgets: a spending limit per agent, with alerts before the cap and a hard stop at it.
- Rate limits per tool: for example, how many emails, refunds or record updates the agent may make per hour.
- Circuit breakers: pause the agent and alert a person when errors, refusals or costs jump.
- Kill switch: one documented action, owned by a named person, that disables the agent's tools or revokes its credentials and sends open work to a human queue. Test it in a drill before go-live.
OWASP's agentic list recommends the same containment for rogue agents: kill switches and credential revocation that can disable an agent at once.
Prompt injection: the risk you cannot filter away
OWASP's 2026 entry explains why prompt injection persists: language models make no architectural distinction between instructions and data. Malicious instructions can arrive through user input, retrieved documents, tool output, audio, images or the agent's own memory. They can be encoded in ways filters miss, including low-resource languages. For an agent, any email, PDF, web page or WhatsApp message it reads can try to give it orders.
Filters help, but the more dependable defence is to limit what a successful injection can do.
- Treat every external input as untrusted, label it as such and keep it apart from your own instructions.
- Keep high-impact tools behind human confirmation, so an injected instruction cannot complete a payment or send data out on its own.
- Restrict where the agent can send data to approved recipients and domains.
- Test with adversarial inputs before go-live and after every change, in Arabic, English, Arabizi and mixed text.
Customer-facing chat and voice agents in Arabic dialects
Customer agents meet the widest range of language. A customer in Riyadh, Amman or Cairo may write in dialect, switch to English mid-sentence or type Arabic in Latin letters. Build the test set from real, redacted conversations across the dialects you serve, and report resolution for each dialect, not only the overall rate.
Voice adds speech-recognition errors on names, numbers and dates. Have the agent read back amounts, dates and reference numbers and get a clear yes before it acts. Define handoff triggers in advance: a complaint, a vulnerable customer, low confidence, repeated misunderstanding, or a request for a person. Pass the conversation history across so the customer does not have to repeat it.
Data protection applies to these conversations. The UAE's Federal Decree-Law No. 45 of 2021 (official Arabic text) names a person's voice among the identifiers that make data personal. Its Article 18 gives people the right, with some exceptions, to object to automated decisions that have legal consequences or seriously affect them. It also requires the controller to provide human review of automated decisions on request. Saudi Arabia, Jordan, Egypt and other countries you serve have their own data protection laws with different rules, and the UAE law's detailed requirements depend on its Executive Regulations. This is general information, not legal advice; your obligations depend on your sector, your data and the countries you serve.
Evaluation before go-live: a checklist
Design sign-off
- A written autonomy specification with the level for each action, signed by the business owner.
- A tool register listing each tool, its permissions and the identity it uses, reviewed by security.
- Confirmation points, budgets, rate limits and the kill switch configured, each with a named owner.
Test evidence
- Task success measured on a fixed test set, with wrong actions counted separately from wrong answers.
- Arabic dialect, Arabizi and mixed-language cases included, with results reported per variety.
- Prompt-injection tests run through every channel the agent reads.
- A kill-switch drill and a handoff test completed, with the audit log reviewed end to end.
After go-live, keep the same test set running. Model updates, prompt changes and new tools can each change behaviour, so rerun the evaluation on every change and review logs and costs on a fixed schedule.
How to start
Our Agentic AI & Chat/Voice Agents service builds multi-step agents with limits, budgets, audit logs and approvals, and chat and voice agents on web, WhatsApp, Teams and phone. AI Operations (AIOps) keeps evaluation, monitoring, cost control and model migration running once agents are live; both sit in our Transformation Lab. AI Security Assessment & Monitoring reviews prompt-injection, data-exposure and permission risks in assistants and agents. The agentic AI and support and voice agents pages show how these fit together.
If you are still deciding which workflows are ready for an agent, start with the free AI Exposure Scan, a 45-minute conversation that ends in a written next step.
Frequently asked questions
What is AI agent governance?
AI agent governance is the set of rules and technical controls that decide what an AI agent may do without a person. It defines which tools the agent can use and with what permissions, which actions need human confirmation, how its actions are logged, how much it may spend and how it is stopped. Good governance is written down before go-live, enforced in code and tested regularly.
What is the difference between an AI agent and a chatbot?
A chatbot answers questions and leaves the action to a person. An AI agent can act: it calls tools to read and change records, send messages, book appointments or process refunds, often over several steps. That ability is what makes agents useful, and it is why they need permissions, confirmation points, budgets and audit logs that a simple chatbot does not.
How do you protect AI agents from prompt injection?
No filter removes the risk completely, because language models do not separate instructions from data. Limit what a successful injection can do. Give agents only the tools they need, use read-only permissions where possible and require human confirmation for high-impact actions. Restrict where data can be sent, and test with adversarial inputs in every language and channel the agent reads, including Arabic and Arabizi.
Should AI agents be allowed to act without human approval?
Only for actions that are low impact and easy to reverse, within set thresholds. Payments above a limit, messages sent outside the organisation, deletions and changes to customer records should need a person to confirm. Set the autonomy level per action rather than per agent, start low, and raise it only when evaluation results and audit logs support the change.
Can AI agents handle Arabic dialects in customer service?
Often, but you only know for your customers by testing on the dialects they actually use. Build an evaluation set from real, redacted conversations in Gulf, Levantine, Egyptian or other dialects, Arabizi and mixed Arabic-English, and report results per dialect. For voice, add read-back of numbers and dates, and clear handoff rules so customers reach a person when the agent struggles.
Get new insights by email
One email when we publish something useful. No spam; leave any time.
Services in this article
Related lab: Transformation Lab ↗
