LLM monitoring in production: a checklist for operations teams
Six signals to watch once an AI assistant or agent is live, who owns each alert, how to move to a new model before the old one is retired, and what to do in the first hour of an incident.
The first sign can be a line on the monthly invoice: the AI assistant cost far more than last month while the number of conversations stayed flat. Or a supervisor notices that Arabic answers have started slipping into English terms after a model update that nobody on the team noticed. Both symptoms mean the same thing: the system changed, and nobody was measuring the part that changed. LLM monitoring in production closes that gap with six signals, each tied to an alert threshold and a named person who acts when it is crossed. We recommend building a fixed test set and an owner list before buying any dashboard tool, because charts without thresholds and owners tend to be looked at and then ignored. The checklist below is ready to copy.
What a cost jump or a quality drop can mean
A cost jump with steady traffic has four causes to check first: longer prompts (someone attached a long policy document to every request), more retries after errors, a switch to a larger model, or abuse. The OWASP Top 10 for LLM Applications 2025 lists abuse of this kind as LLM10 Unbounded Consumption, including "denial of wallet" attacks that run up pay-per-use bills. Its prevention list includes rate limits, user quotas and continuous monitoring of resource use.
A quality drop with no code change points to the model, the documents it retrieves or the users. The provider may have replaced a model version, the knowledge base may have gained outdated files, or customers may have started asking about a product the assistant has never seen. Each cause needs a different fix, so the checklist keeps the signals apart instead of rolling them into one "health score".
An illustrative case: a mid-sized insurer in Riyadh runs a claims assistant in Arabic and English. Its weekly bill rises while claim volumes stay flat. Cost per task, broken down by step, shows that a new prompt template sends the full policy wording with every request. The fix is a prompt change, and no vendor change is needed.
The LLM monitoring checklist: six signals, one owner each
The example alerts below are starting points for LLM monitoring. Set yours from two to four weeks of your own baseline, and write them down.
| Signal | What to measure | Example alert | Owner |
|---|---|---|---|
| Quality | Score on a fixed set of Arabic and English questions with expected answers, run daily and after every change | Score falls more than 3 points below the last accepted run | Product owner of the assistant |
| Cost per task | Tokens and spend per completed task (a resolved ticket, a drafted letter), not per call | 25% above the four-week average for two days running | Budget holder for the use case |
| Latency | Time to first token and total time per task, at the 95th percentile | Above the time agreed with the business, for example 8 seconds for chat | Platform engineer on call |
| Refusals and errors | Share of answers that refuse, time out or fail, split by error type | Refusal rate doubles against last week; any rise in rate-limit errors | Engineering lead |
| Drift after model updates | Model requested versus model that answered; test-set score before and after any provider, prompt or document change | Answering model differs from the approved version | AI operations lead |
| Safety events | Flagged prompt-injection attempts, personal data in answers, actions blocked by guardrails, user complaints | Any personal data in an answer pages the owner at once | Risk or security officer |
Quality on a fixed Arabic and English test set
Build the set from real, redacted questions in the forms your users write: Modern Standard Arabic, Gulf or Levantine dialect, Arabizi and English. Keep it fixed between releases so a change in score means a change in the system. Our guide to Arabic LLM evaluation explains how to build and score such a set, and Agent Evaluation Sets is the service that writes one with you.
Cost per task, not cost per call
An agent can make ten model calls to close one ticket. Measuring per call hides that; measuring per task shows whether a change made the work cheaper or dearer. Tag each request with the use case and the business unit so finance can see who spends what.
Drift and safety events
Record which model you asked for and which one answered. The OpenTelemetry semantic conventions for generative AI define separate attributes for both (gen_ai.request.model and gen_ai.response.model) and reuse the standard error.type attribute for failures. On 22 October 2026 these conventions carry the status "Development" and have moved to their own repository, so expect names to change and pin the version you use. For agents that act in other systems, safety events also include tool calls that were blocked or needed approval; our article on AI agent governance covers those limits.
Who owns each alert threshold?
In LLM monitoring, each threshold needs one named person, a deputy, and a response time. "The AI team" is not an owner: when an alert reaches a group, each member can assume someone else has it. The owner can change the threshold, but only in writing, and reviews it each quarter against the new baseline.
An illustrative case: a telecom operator in Amman runs a WhatsApp support agent. Its risk officer owns the safety alerts, but at weekends those alerts land in a shared mailbox that nobody reads until Sunday. Naming a deputy with phone escalation for Friday and Saturday closes the gap at no tooling cost.
Named owners make it easier to meet what the voluntary NIST AI Risk Management Framework 1.0 (2023) asks for under MANAGE 4.1: post-deployment monitoring plans that include incident response, recovery and change management.
Model migration steps when a provider retires a model
Model retirement is planned work, and it should never arrive as a surprise. Anthropic, for example, states on its model deprecations page that it gives at least 60 days' notice before retiring publicly released models, and that requests to retired models fail. As of 22 October 2026, the same page lists Claude Sonnet 4.5 as deprecated on 30 September 2026 with retirement on 30 November 2026 on the Claude API, and notes that Amazon Bedrock and Google Cloud set their own schedules. Other providers publish their own pages; check the one for the platform you buy through.
- Keep a model register. Every application, the exact model ID it calls, the platform, and the retirement date from the provider's page.
- Name an owner on the day a notice arrives. Put the retirement date in the plan with a switch date at least two weeks before it.
- Run the fixed test set on the replacement. Compare quality, cost per task, latency and refusals against the current model, in Arabic and English separately.
- Check request settings. Anthropic's page notes that API version 2026-10-21 removes temperature, top_p and top_k, so a request that sends any of them returns an error on every model. On earlier API versions, setting them to a non-default value returns an error on Claude 4.7 and later models. A migration, or an upgrade of the API version or to a new major SDK version, can break code as well as change answers.
- Send a small share of traffic first. Watch the six signals for several days before moving everyone.
- Switch with a rollback ready. Keep the old route until the retirement date, then remove it.
- Record the decision and the new baseline. The next set of alerts is measured against the new model.
Incident runbook: the first hour after an alert
- Acknowledge. The owner confirms the alert within the agreed time, or the deputy takes it.
- Classify. Quality, cost, latency, errors or safety. A personal-data leak or an unapproved action goes straight to the risk officer.
- Contain. Switch to a fallback model, lower a user quota, turn off one tool, or route conversations to staff. Pick the smallest step that stops the harm.
- Tell the people affected. The business owner, the support team and, where needed, the users, in Arabic and English.
- Fix and retest. No fix goes live until it passes the fixed test set.
- Learn. Add the failing case to the test set and write a short incident note: what happened, the cause, the fix and the new control.
The NIST framework asks for the same discipline under MANAGE 4.3: incidents and errors are communicated to the relevant people, and the steps to track, respond and recover are followed and documented.
What to log, and what to keep out of the logs
Log enough to explain an alert: the model requested and the model that answered, token counts, time to first token, total time, error type, the use case and a trace that links every step of an agent's task. Full prompts and answers are useful for debugging but can contain customer names, ID numbers and account details. Mask personal data before storing it, keep full text for a short, written retention period, and agree both with your data protection officer before go-live.
How to start
Start LLM monitoring with the one assistant or agent that matters most: write 100 to 300 test questions from its real traffic, and fill in the owner column of the table above. We expect a small team to need about a week for this, and it gives every later alert a meaning. If you want help, our AI Operations (AIOps) service in the Transformation Lab covers evaluation, monitoring, cost control and model migration for AI in production. If the system is an agent, our agentic AI page covers limits, budgets, audit logs and approvals. If you are not sure where to begin, the AI Exposure Scan is a free 45-minute conversation that ends in a written next step.
Frequently asked questions
What is the difference between LLM monitoring and LLM observability?
LLM monitoring tracks a fixed set of signals against thresholds, such as quality on a test set, cost per task, latency, refusals and safety events, and alerts a named owner when one is crossed. LLM observability is the wider ability to explain why something happened, using traces, logs and metrics that link every step of a request. You need monitoring to know that something is wrong and observability to find the cause.
What should an LLM monitoring dashboard show?
Six signals for each assistant or agent: quality on a fixed Arabic and English test set, cost per completed task, latency at the 95th percentile, refusals and errors by type, which model actually answered compared with the approved one, and safety events such as personal data in answers or blocked actions. Each signal should show its alert threshold and the name of the person who owns it.
How often should we re-run the evaluation test set?
Run the fixed test set daily on a schedule and again after every change: a new prompt, new documents in the knowledge base, a new model version or a provider notice. Daily runs catch changes nobody flagged, such as new files in the knowledge base or a model alias that now points to a newer version; change-triggered runs stop a bad release before users see it. Keep the set fixed between releases and add every real failure to it after an incident.
What happens when a provider retires the model we use?
Requests to a retired model fail. Anthropic, for example, states it gives at least 60 days' notice before retiring publicly released models, and partner platforms such as Amazon Bedrock and Google Cloud set their own schedules. Keep a register of model IDs and retirement dates, test the replacement on your fixed test set, send a small share of traffic first, and switch with a rollback ready.
Get new insights by email
One email when we publish something useful. No spam; leave any time.
Services in this article
Related lab: Transformation Lab ↗
