Copilot Code Review: evidence to require when AI runs code

A review comment can describe a plausible defect and still be wrong about actual behavior. The cost appears when a team changes correct code, dismisses a real risk or mistakes a closed conversation for a completed check. Copilot Code Review brings an engineering question into focus: what evidence do we need before acting on an AI observation?

On September 11, 2026, GitHub announced broader shell tools for code review, including builds, tests and scripts, behind the agent firewall. It also announced automatic resolution of addressed comments during rereview and an ensemble of agents for Lite. That update is our starting point; the criteria below are our implementation proposal, not results measured by Wasyra.

The useful mechanism is a short loop: form a hypothesis about the diff, choose a check that could disprove it, execute it in a bounded environment and tie the result to the reviewed code. Console output is useful only when we know which command produced it, against which revision and with which data. Asking the agent to be more careful leaves too many decisions implicit.

The documentation states that agentic review capabilities use GitHub Actions. The available environment therefore matters: a checkout without dependencies or fixtures can prevent a valid test. Before expanding the pilot, prepare a reproducible installation path and confirm that the command actually discovers the expected tests. An execution that exits without errors but runs no cases does not demonstrate the behavior under investigation.

Record the reviewed SHA, comparison base, affected files and the scenario that triggers the suspected defect. If another commit arrives, earlier evidence remains historically useful but does not automatically validate the new state. To decide what to repeat, determine whether the function, its dependency, the fixture or the configuration supporting the test changed.

The record can be small: one observation, one command, an expected result and an observed result. Include the exit code and the number of cases executed. If a dependency is missing, classify the finding as awaiting verification. Do not turn an inability to reproduce it into a claim that the defect does not exist.

A test script is executable code too. For the pilot, use synthetic data, narrowly scoped credentials and disposable services. Define what the environment needs to download and who approves expanding that access. Review installation and setup scripts as carefully as test commands; familiar names do not guarantee harmless effects.

GitHub documents specific firewall limits: it does not directly cover MCP server processes or configured setup steps, and its scope is limited to the GitHub Actions appliance environment. The documentation also warns about potential bypasses. Review those paths separately and retain network restrictions as one layer of the design, without assuming complete isolation.

Hypothetical example: an API must prevent a user from reading documents belonging to another organization. The agent notices that a query filters by document identifier but appears to omit the organization. A useful test prepares two fictional organizations, attempts to read the other organization’s document and checks for denial. A successful build does not answer that authorization question.

Run the same case against the previous state and the proposed fix. If the test passes in both, another authorization boundary may already exist, or the case may not reach the vulnerable path. If it fails in both, the fix may be incomplete. Add the allowed same-organization case to detect a fix that simply blocks everyone. Keep results free of real customer data.

Use three states in the team process: comment addressed, behavior verified and change accepted by the owner. Each answers a different question. The first organizes the conversation; the second requires evidence; the third includes compatibility, scope and operational consequences. This separation prevents a tidy interface or a convincing explanation from replacing the technical decision.

GitHub’s current guide distinguishes Comment reviews from approvals and allows Copilot approvals to be configured. Check the repository’s actual policy. Retain the required checks appropriate to the product and define who can accept exceptions. A local test run by the reviewer and a required CI check have different provenance and responsibilities.

Select historical changes with confirmed defects, known false positives and clean cases. Keep a subset out of the instructions used to prepare the agent. Evaluate whether it finds the defect, supplies a valid reproduction and reduces the human work needed to decide. Count missed defects as well: measuring only accepted comments favors a reviewer that says little and overlooks problems.

Separate results by change type: authorization, business logic, interface or configuration. Record total time, consumption, failed commands and reviews left without evidence. Set acceptance criteria before seeing results and repeat a sample to detect variation. A vendor’s promotional figures do not replace this evaluation in your repositories.

Adoption makes sense when the team can reconstruct the environment, constrain access and audit results. If tests depend on production or nobody maintains the fixtures, start by fixing that foundation. Automating a fragile check can multiply uncertainty and consume time without improving the merge decision.

The five-criterion contract supports adoption of this update with verifiable expectations. We have not run a Copilot benchmark for this article and do not claim a reduction in defects. If you are introducing AI review, start with one critical path and one reproducible test. Wasyra can help define that pilot as part of a custom agent project, with agreed scope and acceptance conditions.