Copilot Auto: evaluating cost and quality for your team

On September 14, 2026, GitHub announced three preferences for Copilot Auto: efficiency, balance and intelligence. They steer selection toward cost, balance or quality, respectively. The rollout covers VS Code, Copilot CLI and the Copilot app; billing still depends on the selected model.

For an engineering team, the useful question is which work suits each preference and what evidence should justify accepting the result. Changing a setting is quick; discovering that apparent savings disappeared into reviews, corrections or incidents takes longer. Design the experiment before applying one preference across the team.

Auto documentation explains that routing considers complexity and availability within the models allowed by the plan and policies. Choosing intelligence does not necessarily pin a large model for every request. The tier name alone therefore does not constitute a reproducible technical specification.

Our recommendation is to keep three decisions explicit: which capability to request, which tools the agent may execute and who accepts the change. A task that is cheap to generate can have expensive consequences, such as changing an authorization rule. Repository owners should retain control over secrets, publication and data changes regardless of the selected tier.

Select small, medium and difficult tasks from your own repository. Include a localized fix with an existing test, an integration spanning several modules and an investigation of a non-obvious failure. Prepare context, acceptance criteria and a common baseline. Comparing a detailed request with an ambiguous one would wrongly attribute the whole difference to routing.

When comparing preferences, use independent branches or environments so an earlier solution does not reveal the answer to the next attempt. Alternate execution order and retain failures. A sample containing only the best results may produce an attractive conclusion that is useless for capacity planning. Also record external dependencies or permissions that prevented completion.

Prepare a reference outcome for every task: expected behavior, minimum evidence and conditions that require rejecting the change. When feasible, hide the selected preference from reviewers until they have assessed the delivery. This helps reduce the temptation to accept a solution because it came from the tier the team considers more capable. A large laboratory is unnecessary; agreeing on the rules before seeing results is essential.

Set a time or attempt limit and record tasks that exceed it. Quietly excluding failures artificially improves any average. If the sample is small or differences are unstable, document the uncertainty and expand the experiment before changing policy for the entire team.

Define the comparison unit as an accepted task supported by evidence, and record five criteria for every attempt. Keep the data simple enough for another reviewer to reconstruct the decision. If understanding the pilot requires a complex new tool, the team is probably measuring too many things at once.

Consider a hypothetical case: an inventory screen duplicates a stock movement when a user retries a request. The task might look like a button adjustment, but it may involve server-side idempotency. Before evaluating models, require reproduction of the failure, identification of where duplication occurs and proof that the fix withstands equivalent requests.

If one attempt disables the button while another protects the complete operation, they are not equivalent outcomes even if both pass a superficial test. Evaluation must detect that difference before adding up costs. Check whether the agent changed unrelated modules or introduced unnecessary dependencies. The pilot should reveal how real work gets solved rather than reward the first convincing answer.

A separate official update on September 11 adds activity metrics for the dedicated VS Code Agents window. GitHub clarifies that this scope differs from editor Agent Mode. It helps explain usage but does not itself demonstrate that delivered changes are correct or that delivery cycles have shortened.

Connect aggregate activity to task and review outcomes without turning message counts into employee rankings. One team may send fewer messages because it prepared better context; another may send more because it is solving a new problem. State what each indicator includes, excludes and leaves missing to prevent misleading comparisons between teams.

At the end, adopt a preference for a task class only when the evidence supports the change. Retain stronger review for sensitive operations and an escalation path when an attempt stalls. Document exceptions: an architectural investigation may require different treatment from a mechanical edit in the same repository.

Repeat a sample when available models, team policies or the nature of the work change. Preserve earlier measurement context so you know which comparisons remain valid. The responsible conclusion is not that one tier always wins: it is that the team can explain what it accepts, the complete effort required and the signals that would trigger reconsideration.