Continuous agent evaluation: nine release gates

An agent can preserve latency, availability, and technical error rates while degrading the outcome users receive. A source changes schema, the router selects another sub-agent, a tool description becomes ambiguous, or a model changes behavior; the answer still arrives fluently and the traditional monitor stays green. On October 16, 2026, AWS published a nine-stage framework for operating that risk with evaluation from the first contract through live traffic. The news was two Lima calendar days old when consulted on October 18. It is evidence of one provider's practice and implementation, not a universal quality guarantee.

The CTO decision is not to buy nine tools. It is to close one loop: define success, compare candidates, block regressions in CI, approve the release, observe canaries, attack boundaries, detect drift, maintain an operational pulse, and turn confirmed failures into permanent cases. That loop must operate at three levels: tool, agent, and orchestration. If only the final response is scored, a correct number can hide the wrong tool; if only tools are tested, the orchestrator can combine incompatible results. Quality is a property of the system and its trajectory.

Start with the business decision the agent must support and translate it into observable failures. A financial assistant does not need an abstract 'quality' label: it needs to distinguish an unsourced figure, wrong currency, wrong period, confused entity, unauthorized calculation, and out-of-scope response. Each class needs severity, ownership, evidence, and a promotion threshold. Then add safety, performance, cost, and operational control. A candidate can improve average accuracy and still be unacceptable if it broadens permissions, doubles p90, or makes it impossible to reconstruct which tool produced a claim.

The dataset is also part of the contract. Separate a small sentinel set for every commit, a broader golden set for the release gate, and adversarial cases that try to force data leakage, forbidden tool selection, or scope expansion. Stratify by risk, language, user type, source, length, and tool path. Preserve initial state, expected responses or assertions, allowed trajectory, and data version. AgentCore's on-demand evaluation API accepts OpenTelemetry traces and can receive expected responses, assertions, and expected tool trajectories; that design illustrates why a simple prompt-response pair is insufficient for an agent.

AWS's sequence is useful as a map, but it must be adapted to each product's risk and cadence. Foundation defines what 'good' means; Experiment compares versions; Merge finds fast regressions; Gate decides whether the candidate deserves production; Canary looks for post-deploy failures; Adversary pushes safety and quality boundaries; Drift compares segments and time; Pulse supports diagnosis now; Feedback turns validated incidents into changes and new cases. They are not nine independent ceremonies: they share datasets, taxonomy, versions, and traces.

A probabilistic judge helps with semantic properties such as grounding, relevance, or completeness, but it should not replace an available oracle. Use code for schemas, permissions, amounts, identifiers, SQL, invariants, exact tool selection, and absence of known secrets. Use human review for domain ambiguity, severity, and editorial quality. Reserve the LLM judge for criteria whose rubric can be written, exemplified, and audited. AgentCore documentation distinguishes built-in, third-party, custom LLM, and custom code evaluators; the right choice depends on the property being tested, not dashboard convenience.

Calibrate every judge against a set labeled by qualified people and measure false positives, false negatives, and per-class agreement, not only global correlation. Version the model, prompt, scale, and examples; recalibrate when any changes. Keep boundary cases outside the set used to tune the rubric. When the judge disagrees with a deterministic oracle, the oracle governs. When it disagrees with humans, do not average away the conflict: revisit the definition. A continuous score looks precise, but it can compress severe errors and stylistic preferences into the same decimal.

Imagine an agent that prepares renewals: it queries CRM, billing, tickets, and usage, then drafts a summary and recommends an action. The team changes the billing connector and every call still returns HTTP 200. However, the currency field is now empty for a Latin American cohort. The agent fills in the symbol from the country and produces plausible figures. Availability, latency, and errors remain normal; quality degraded without a visible outage.

The contract would have caught the problem at several layers. A tool test requires source-provided code and currency, not inference. The CI sentinel contains two multi-currency accounts. The release gate replays validated responses and compares candidate with production. The canary sends a synthetic query after deployment. Drift segments by region and detects a grounding drop even when the global average barely moves. Pulse links the alarm to the trace and connector version. Feedback adds the incident to the golden set, but the proposed fix passes the nine gates again; it is not promoted merely because an automated recommendation sounds convincing.

This example is hypothetical. It does not describe a client, measurement, or Wasyra experience. Its purpose is to show that the same failure should be detectable before and after deployment, and attributable to a specific tool without indiscriminately storing prompts, tickets, or customer data.

Online evaluation needs deliberate sampling. Combine a random sample for trend detection with oversampling of critical paths, new versions, sensitive tools, long sessions, and cohorts where the model has less evidence. Add stable synthetic traffic to distinguish system drift from changes in the user mix. AgentCore's documented online configuration supports trace sources, filters, sampling rate, and up to 25 evaluators. That limit is a technical capability; it does not imply that enabling more evaluators produces better signal.

Record version identifiers, trajectory, normalized outcome, and required scores with explicit retention and access. The earlier article on agent telemetry explains why observability does not require copying complete content. For an investigation, enable content only in a bounded, authorized window. Also preserve a holdout that does not feed recommendations: if every evaluated failure immediately tunes prompts or tools, the system can optimize for the judge without improving real work.

Week one: choose one high-value, narrow-scope flow; define five to ten failure classes, severity, and a small domain-reviewed golden set. Week two: instrument model, prompt, data, and tool versions; add deterministic oracles and calibrate only the semantic judges you need. Week three: connect a sentinel to CI and a broader replay to release; require comparison against the current baseline and block any safety violation even if the average improves. Week four: deploy a canary, synthetics, risk-based sampling, alarms with a runbook, and a practiced rollback.

Do not automate feedback until the team can explain false positives, reconstruct a trace, and roll back a version. Prompt or tool-description recommendations are candidates, not approved changes. Review new failures and coverage weekly; retire redundant cases monthly without deleting high-severity incidents. Measure cost per evaluated session, time to detect, time to attribute, regressions blocked, and escaped failures. A single score works for a summary; the decision needs the distribution by class and cohort.

The value of the nine-stage framework is not a longer list of metrics. It connects definition, release, live traffic, and incidents under the same versioned contract. Start with one flow, three evidence levels, and a few failures the business actually understands. Combine deterministic oracles, calibrated judges, and human review. Then expand coverage according to incidents and risk, not according to how many evaluators are available.

If you need to take an agent from demo to an operable system, Wasyra can help design datasets, tools, permissions, traces, and promotion gates. The goal is not to declare an agent 'trustworthy' once: it is to prove which version passed, which degradation was detected, and how the team recovered service without hiding uncertainty.