Agent telemetry: observe without capturing secrets

Observe the execution structure before its content. Operating an agent usually requires knowing which model answered, how long the session lasted, how many turns and tool calls occurred, which kind of tool failed, whether policy blocked an action, and whether a change was accepted or reverted. Identity can be added as a separate class when a valid audit purpose exists; raw prompts, responses, arguments, results, and paths should remain off until a specific investigation justifies each field. GitHub formalized this separation on October 12, 2026: its managed telemetry can attribute activity to an account, local user, and machine without enabling conversation content. The change does not prove compliance or security on its own. It does provide a useful architecture: minimize by default, govern every class independently, and send signals only to an approved collector with explicit access, retention, and deletion rules.

The central news was two calendar days old in America/Lima when consulted on October 14. GitHub documents that identity and five content classes are off by default and that managed configuration takes precedence over local settings. The VS Code guide confirms that model names, tokens, durations, counts, hook outcomes, and acceptance metrics remain available without content capture. Those are provider contract claims; this article does not present an independent audit of every client or assume that every harness emits exactly the same attributes.

An agent trace has several layers. The invocation span wraps the complete session; model calls, tool executions, and hooks appear beneath it. Without copying text, each layer can retain structure: provider and resolved model, input and output tokens, time to first token, total duration, tool name or type, success or error, hook decision, and turn count. It can also emit aggregatable metrics for calls, latency, edit acceptance, and change survival. That structure answers operational questions without reconstructing the conversation.

The new separation turns capture into six conceptual switches. Identity adds the authenticated account, operating-system user, and host name. Prompts can include user, system, and developer instructions, workspace context, attachments, compaction summaries, and skills. Responses can include output and supported reasoning. Tool arguments and results can reveal commands, files, schemas, errors, and MCP or plugin identifiers. Policy detail can expose complete sandbox paths. A field being useful for debugging does not automatically make it necessary for continuous operations.

The design principle is to separate attribution from inspection. Knowing who started a session or which team sees concentrated errors does not require storing the prompt. Understanding which tool fails may require only its type, outcome, and duration; raw arguments enter only an approved diagnostic window. Even identity should be challenged: a pseudonymous identifier or team dimension may answer a capacity question without turning the dashboard into individual surveillance. Every field needs a purpose, owner, consumer, retention period, and deletion test.

Do not start by asking what the tool can export. Start with the decision a signal will enable. The following five decisions form a minimum contract; if one lacks a verifiable answer, capture should remain off or stay inside an isolated pilot.

Imagine a SaaS company that lets an agent read tickets, inspect repositories, run tests, and propose pull requests. The platform team wants to answer three questions: what share of sessions reaches an accepted change, which tools concentrate errors, and which user started a run that modified a regulated repository. The easy path would store every prompt, command, and result. That design would duplicate tickets, code, local paths, and potential secrets inside the observability backend.

The hypothetical pilot starts with content-free spans and metrics. It keeps session ID, agent type, model, normalized repository, initial commit, tool category, outcome, duration, policy block, and acceptance decision. Named identity is enabled only for the regulated group, with audit-team access and short retention; other teams use pseudonymous identifiers. Arguments, outputs, and raw policy detail remain off. A separate map links known failures to normalized error codes so investigators do not need complete stderr.

The incident exercise introduces three controlled failures: a test-runner timeout, an attempt to read a blocked path, and human rejection of the diff. The team must locate the phase, session, and authorized owner without reconstructing the ticket or seeing file content. It then verifies that expiry removes identities, that an analyst without the role cannot query the regulated group, and that no event contains planted tokens. This example is not a Wasyra customer or result; it shows how to make privacy a testable property of the pipeline.

Evaluate three outcomes separately. Operational utility: time to locate the failing stage, share of diagnosable incidents, pipeline latency, and cost per session. Agent outcome: changes accepted, rejected, or reverted, errors by tool type, end-to-end duration, and change survival. Exposure: sensitive fields per event, identities accessible by role, sessions beyond retention, samples containing planted secrets, and unapproved destinations. More detailed telemetry wins only when it improves a concrete decision without worsening the third group.

Add negative tests to the release. Run prompts containing a canary secret, sensitive filenames, profile paths, and a tool argument shaped like a token. Inspect the payload at the client, collector, and backend; testing only the dashboard misses data discarded after ingestion. Also change a local preference to try enabling content and confirm that effective managed policy blocks it. Repeat across CLI, app, and every VS Code harness in use because fine-grained support is not uniform.

Record the policy version with every session. If managed configuration replaces the complete object rather than merging it, a partial update can remove a restriction without the dashboard revealing it. Rollout evidence should include the expected file, observed effective configuration, minimum supported client, a test event, and the query proving retention. Without that chain, policy is written but not necessarily enforced.

Turning content off reduces exposure but does not anonymize telemetry. A login, local username, and hostname identify a person or device; a repository URL, branch, or commit can reveal projects; a tool name or error code can be sensitive in some contexts. Treat that metadata according to its actual classification and applicable rules. This article offers engineering criteria, not legal advice or an automatic basis for employee monitoring.

Do not confuse observability with semantic evaluation either. Token counts, duration, and diff acceptance do not prove that an answer is correct, safe, or useful. Keep quality datasets and reviews separate, using authorized and minimized examples. If an investigation requires content, use a time-bounded window, an explicit cohort, and recorded approval; do not turn an exception into the new default.

Finally, verify coverage. GitHub's reference documents differences across Copilot CLI, the app, Agent Host, and VS Code's local harness. Central policy cannot compensate for an old client or a surface that supports only the legacy switch. Inventory versions, block incompatible configurations where appropriate, and treat any session without confirmed posture as a separate path rather than equivalent telemetry.

Agent telemetry matures when structure, identity, and content stop being an inseparable bundle. Start with operational spans and metrics, add attribution only to the perimeter that needs it, and enable content only for a bounded investigation. The five gates—purpose, minimum, boundary, enforcement, and outcome—turn that intent into evidence that can be audited and reversed.

If you need to design a pilot with tools, permissions, traces, and evaluations, Wasyra's custom agent service can help turn these gates into an operating contract. It is an implementation service on wasyra.com, separate from the Agents product available at agents.wasyra.com. The goal is not to observe more; it is to retain enough signal to act without creating another risk surface.