GitHub introduced Copilot Evals in public preview on November 17. It creates, runs and compares evaluations for skills, custom agents and MCP tools. It uses the local Vally engine, with each run approved before execution. Versions are announced for VS Code, Copilot CLI and the app, for Individual, Business and Enterprise users.

The official repository describes saved reports and Evals Canvas for inspecting tasks and results. The Marketplace listing specifies local Node.js stdio MCP support, one server per run with a fresh start. Python and remote MCP servers are not currently supported. A general statement that the product evaluates MCP therefore does not cover every integration type.

What a local task should measure

A useful evaluation describes both the desired output and permitted changes. Our proposed example uses a demonstration project in which an agent edits one configuration file and creates a report. Passing requires correct content and no changes to unrelated files. A polished report accompanied by an incorrect side effect should remain a failure.

Preserve the same initial state for each comparison. If the next run receives easier inputs or an already repaired project, improvement cannot be attributed solely to the updated skill. Recording the task, inputs and configuration lets another person understand what is actually being compared.

Check incomplete outcomes too

Include a missing file or temporarily unavailable tool. The expected response might be to preserve partial work and clearly identify the blocker. This checks whether the agent distinguishes completion from an attempt. Real customer data is unnecessary merely to make a demonstration look convincing.

Repetition matters when behavior varies between runs. A single success does not erase an observed failure. A reviewer should identify which particular error would block deployment and why it matters to the product. Establish those criteria before running the evaluation rather than changing them after an inconvenient result.

Keep preview limits visible

We have not installed the extension or performed these examples. They describe a proposed evaluation method, not evidence of the product's effectiveness. The preview provides tools for a more inspectable discussion of agent quality. Its value to a particular team depends on whether the tasks expose that team's real failure modes and whether saved evidence supports a meaningful review.