Most agent dashboards measure the wrong thing. Latency, token count, and error rate tell you whether the agent ran. They say nothing about whether it was right. An agent can complete every request in three seconds with a 0% error rate and still hallucinate a policy that does not exist. AWS calls this a quality failure, and it is invisible to every operational metric you already have. The June 2026 AWS post on debugging production agents shows the tell directly: a session burning 266,900 tokens across 177 spans with a 0% error rate. The dashboard was green. The agent was stuck in a loop.

So we build two layers. The first is the operational layer: a Strands agent instrumented with the AWS Distro for OpenTelemetry, emitting traces, token metrics, and latency into the CloudWatch GenAI Observability dashboard with almost no code. The second is the layer nobody ships: a quality signal. We sample the agent's live responses, score each one with a cheap judge model on Bedrock, push that score to CloudWatch as a first-class custom metric, and attach an alarm that pages you over SNS when the rolling average drops below a threshold.

The non-obvious design choice is treating a model-graded score as a metric, not a log line. Once quality is a CloudWatch metric with a name and a namespace, everything CloudWatch already does for CPU and latency works for correctness: alarms, dashboards, anomaly detection. What you ship at the end is an agent whose regressions page you before your users file a ticket.

You need an AWS account with Amazon Bedrock model access enabled for at least two models: one for the agent (this tutorial uses Claude 3.7 Sonnet) and one cheaper model for the judge (Amazon Nova Lite). Turn both on in the Bedrock console under Model access before you start, or the first call returns AccessDeniedException.

You need Python 3.10 or newer, the AWS CLI configured with credentials (aws configure), and permission to create CloudWatch alarms, put custom metrics, create an SNS topic, and enable CloudWatch Transaction Search. You should be comfortable reading Python and have seen the Bedrock Converse API before (episode 1 of this series covers it if you have not).

One account-level switch matters: CloudWatch Transaction Search must be enabled once per account before any agent traces show up. We do that in setup. Everything else is code you run locally, so there is no infrastructure to stand up and no container to build.

Create a project and install the dependencies. The [otel] extra on Strands and the aws-opentelemetry-distro package are what make instrumentation automatic.

Enable CloudWatch Transaction Search once for the account. This routes OpenTelemetry spans into CloudWatch Logs so the GenAI dashboard and Transaction Search can read them. The resource policy lets X-Ray write spans, and the second command points trace segments at CloudWatch Logs.

Give it up to ten minutes to take effect. You can index 1% of traces at no cost; adjust the sampling percentage later with aws xray update-indexing-rule if you need more. Confirm it is on before sending traffic:

The response should show "Destination": "CloudWatchLogs" and "Status": "ACTIVE". If it still says PENDING, wait and re-run.

Write the agent. This is an ordinary Strands agent with two tools. Nothing here is observability code. The instrumentation comes from how we run it, not from what we import.

The termination clause in the system prompt is not decoration. The AWS debugging guide traces most infinite loops to prompts that tell the agent to keep trying with no stop condition. A prompt that says "never give up" is how you get a 177-span session. Give the model an explicit exit.

Now run it under the ADOT auto-instrumentor. These environment variables tell OpenTelemetry to use the AWS distro, export over OTLP, and stamp the telemetry with a service name so it groups under one agent in the dashboard.

The opentelemetry-instrument wrapper loads the config from those variables, auto-instruments the Strands runtime and every Bedrock call underneath it, and ships spans to CloudWatch. You wrote zero lines of tracing code. Open the GenAI Observability dashboard in the CloudWatch console, and under the Bedrock AgentCore tab you get an Agents view, a Sessions view, and a Traces view with token usage, latency, and the full reasoning trajectory per request.

Before adding anything, learn to read what you already have, because the operational metrics catch two of the three failure classes on their own. AWS groups agent failures into quality, reliability, and efficiency. Reliability shows up as error rate. Efficiency shows up as latency and token count. Both are already on the dashboard.

The trap is quality, which is why the token metric is worth a second look. High token usage with a low error rate is the signature of a silent loop: the agent is running, not crashing, and cannot finish. This CloudWatch Logs Insights query, run against your agent's log group, surfaces the offending sessions.

Take the worst SessionId and pull its trajectory to confirm. A healthy conversational turn is a handful of spans and one to five seconds. A session with 177 spans and 85 seconds of latency is looping, and the trace waterfall will show the same tool call repeating with near-identical inputs.

This is the ceiling of what operational telemetry buys you. It catches loops, tool errors, and latency regressions. It does not catch an agent that answers quickly, cleanly, and wrong. For that you need a different kind of signal.

Here is the piece that closes the gap. After the agent answers, we ask a second, cheaper model to grade the answer, then push that grade into CloudWatch as a metric. This is the online-eval pattern, and the important move is the last line: a model-graded score becomes a named metric in a namespace, which means CloudWatch can alarm on it exactly like it alarms on latency.

Two decisions carry weight here. The judge runs at temperature 0 so the score is stable across identical inputs, and it returns strict JSON so parsing does not depend on prose. The judge model is Nova Lite, not the agent's Sonnet, because grading is a narrow classification task and running it on a frontier model at sampling scale is money set on fire.

You do not grade every request. Sampling a fraction keeps judge cost low while still giving CloudWatch enough data points to average. Wire it into the agent loop and score, say, one in five turns.

Every sampled turn now drops a JudgeScore data point into the SoftwareLetters/AgentQuality namespace. In the CloudWatch Metrics console, browse to that namespace and you will see the scores plotting over time, per agent. Quality is now a time series.

A metric you have to remember to look at is not monitoring. Attach an alarm so a drop pages you. First an SNS topic to deliver the page, then the alarm itself.

The alarm fires when the average judge score sits below 3.5 for three consecutive five-minute windows. The three-window requirement is deliberate. A single bad answer should not page you at 2 a.m.; a sustained fifteen-minute slide means a prompt change, a model swap, or a data-source shift degraded quality, and someone should look now. TreatMissingData="notBreaching" keeps quiet periods with no sampled traffic from flapping the alarm.

Confirm the subscription email SNS sends, then break the agent on purpose to watch it trip. Point the refund_policy tool at the wrong window (change <= 30 to <= 3), send a batch of refund questions, and the judge will start scoring the now-wrong answers low. Within about fifteen minutes the alarm moves to ALARM and SNS emails you. That is a quality regression caught by a metric, not by a customer.

Operators want one screen, not three consoles. put_dashboard stitches the operational metrics and the quality metric into a single view. This widget plots the average judge score next to where you would add token and latency widgets from the bedrock-agentcore namespace.

Open the AgentHealth dashboard and the correctness trend sits beside your reliability and efficiency numbers. That is the whole point: quality is no longer a separate investigation you run after a complaint, it is a line on the same chart as latency.

Four checks confirm the whole pipeline. First, run opentelemetry-instrument python agent.py, then open the GenAI Observability dashboard's Bedrock AgentCore tab; within a few minutes support-agent appears in the Agents view with token and latency metrics and at least one trace in the Traces view. Second, run run_with_eval.py a dozen times and browse to the SoftwareLetters/AgentQuality namespace in the Metrics console; you should see JudgeScore data points averaging 4 to 5 on correct answers. Third, run alarm.py and confirm the alarm shows state OK under Alarms after a healthy batch. Fourth, introduce the deliberate bug from Step 4, send ten wrong-answer questions, and watch the alarm flip to ALARM and the SNS email land. If all four hold, you have operational and quality observability working together.

If the agent produces no traces, Transaction Search is almost certainly not active yet. Run aws xray get-trace-segment-destination and confirm Status: ACTIVE; a fresh enable takes up to ten minutes, and spans only start flowing after it flips. If you see AccessDeniedException on the first Bedrock call, the model is not enabled in Model access for your region, or your caller lacks bedrock:InvokeModel; enable both the Sonnet and Nova Lite model IDs and retry.

If judge() throws a JSONDecodeError, the judge model wrapped its JSON in prose. Keep temperature at 0 and tighten the rubric to "reply with only JSON," or strip everything before the first { and after the last } before parsing. If the alarm never leaves INSUFFICIENT_DATA, you are not sending enough sampled data points inside the period; lower the sampling threshold from 0.2 or shorten the alarm period while testing. If the alarm flaps on and off during quiet hours, that is missing data being treated as breaching; keep TreatMissingData="notBreaching". Finally, if judge cost climbs, you are sampling too high or grading on too large a model; Nova Lite at a 10 to 20% sample rate should keep the eval bill well under the agent's own inference cost.

Following along runs a few dollars at most. CloudWatch Transaction Search indexes 1% of traces free, and 1% is plenty for a test. Custom metrics are 0.30 dollars per metric per month, and you are creating one. Alarms are 0.10 dollars each. The real variable is Bedrock inference: a few hundred agent turns on Sonnet plus a 20% judge sample on Nova Lite is well under 5 dollars. Delete the alarm, the custom metric, the SNS topic, and the dashboard when done, and turn Transaction Search back off in the CloudWatch console if you enabled it only for this walkthrough, since leaving it on keeps indexing production traces.

Three extensions, easiest first. Swap the homegrown judge for AgentCore Evaluators, the managed service AWS shipped for exactly this: it inspects agent sessions in real time and scores them against quality criteria without you running a judge loop, which removes the sampling and JSON-parsing code entirely. Next, add a second custom metric for tool-selection accuracy by having the judge also score whether the right tool was called, then alarm on that separately, since a wrong-tool regression and a hallucination regression usually have different root causes. Hardest, export the same OpenTelemetry stream to Datadog, Grafana Cloud, or Elastic by changing the OTLP endpoint, so the quality metric lands in whatever your team already watches at 3 a.m.

The reframing worth keeping: an error rate tells you the agent broke, and a token count tells you it wasted money, but only a graded score tells you it was wrong. If correctness is the thing you actually sell, it should be the thing you alarm on. Right now, for almost everyone shipping agents, it is the one number missing from the dashboard.