AI training preferences in robots.txt

On October 30, 2026, three calendar days before this publication in America/Lima, OpenAI and Google announced plans to honor the shared __usage_ai-training token. The Internet-Draft defining how an ordinary robots.txt group can express a preference for or against using a resource in generative training was published that same day. The concrete decision for a founder or CTO is not to copy two lines and declare the problem solved. It is to turn editorial and legal intent for each content family into a versioned policy, test how that policy coexists with provider-specific controls, and measure which consumers honor it. The token reduces duplication for compatible participants; it does not make a public resource private, evict a crawler by itself, or prove that a historical dataset was deleted.

The development is still an announced adoption, not universally deployed coverage. OpenAI says it aims to implement the signal in the coming months while keeping its existing controls. Google targets early 2027 and says that, when no explicit Google-Extended rule exists, it plans to interpret the shared opt-out as an equivalent disallow for that control. The Internet-Draft expires on May 3, 2027 if it does not progress and may change during IETF discussion. A rollout should therefore record the version, date, and observed behavior rather than treating the draft as a finished standard.

The draft reuses RFC 9309 grammar and path matching. A group with User-Agent: __usage_ai-training and Disallow: / expresses a preference against training for the whole site; a more specific Allow can open a path such as /press-kit/. If the token is absent, no preference is expressed through this mechanism. The User-Agent: * wildcard group must not be interpreted as matching the token. This separation matters because existing parsers can ignore the unknown group without changing their crawling behavior.

The key word is use. The draft defines AI training as using an asset to modify the learned parameters of a generative model. Classification, ranking, and scoring fall outside that category even when the system produces a rationale. The same robots.txt file can contain a usage preference and crawler traffic rules, but they are orthogonal planes. If you need to prevent access, protect secrets, or limit load, keep authentication, authorization, rate limiting, crawler-specific rules, and infrastructure controls.

Hypothetical example: a company publishes documentation, a blog, commercial pages, and an authenticated portal. It wants the blog and commercial pages to remain visible in search and eligible to support linked answers, but it does not want to express consent for generative training. It does allow training with a licensed press kit. The decision matrix should first classify the purpose and sensitivity of each path; only then should it produce the shared rule, provider exceptions, and access controls. Starting with syntax reverses the problem and usually mixes incompatible goals.

The minimum implementation would deny training on / and allow /press-kit/, while preserving independent crawler rules and protecting /portal/ behind identity. The company still has to decide what happens across subdomains, CDNs, assets, feeds, regional copies, and content distributed by third parties. robots.txt is resolved per origin; a rule on www.example.com does not automatically govern assets.example.com or a partner replica. The policy should also define who approves exceptions and which contract prevails when a private agreement exists, because the draft allows private agreements to override the public preference.

Treat the change as a policy migration, not an isolated edit. The content owner should approve the usage matrix; legal should review contracts and jurisdictions; platform should generate and serve valid rules; SEO should confirm that intended indexing still works; security should verify that no sensitive path relies on robots.txt; and observability should preserve a snapshot of the effective file and provider responses. Each gate should produce dated evidence and a clear rollback condition.

Do not immediately remove proprietary tokens. During the transition, publish the shared rule alongside documented controls such as Google-Extended or OpenAI crawlers when the intent is equivalent, and declare every exception explicitly. Google expects a specific Google-Extended rule to take precedence over its interpretation of the general token. That precedence is useful, but it also creates a risk of contradictory configuration. An internal linter should expand paths and providers to show the effective result before deployment.

The first test is deterministic: fetch robots.txt from every origin as an external client, preserve headers and hash, parse the token group, and evaluate a table of positive and negative paths. Include trailing slashes, casing, escapes, parameters, and longest-match behavior. Repeat the test from the CDN and after every infrastructure change. A correct file in the repository does not prove that the effective version reached the edge.

The second test observes effects without inventing causality. Segment logs by verified user agent when a provider publishes a verification method, but remember that the policy token may never appear as an HTTP request user agent. Measure changes in crawling, indexing, referrals, and cited appearances separately. If traffic falls, do not conclude that the training opt-out caused it: the draft explicitly says the usage signal is independent of crawler rules. Open an incident only with a reproducible diff between policy, expected behavior, and observed behavior.

The third test is governance. Review the draft version, participating providers, implementation dates, and definitions every quarter. Preserve snapshots of the documentation behind the decision; a future promise should not be recorded as active support. If a provider changes the relationship between training, grounding, and search, rerun all six gates before adjusting the rule.

robots.txt is public and declarative. A non-cooperating actor can ignore it; a user agent can be spoofed; a copy obtained before the change may remain under a different policy; and a distributor may serve the same content from another origin. The announcement also provides no retroactive audit, universal per-resource receipt, or deletion mechanism. If data is confidential, personal, or contract-bound, the correct control sits before download. If the organization must prove compliance, it should combine signaling, contracts, distribution inventory, verifiable records, and a dispute channel.

There is commercial risk as well. Indiscriminately blocking crawling or grounding in an attempt to prevent training may reduce discovery and referrals without improving the intended control. The opposite mistake is accepting every form of use out of fear of losing SEO. The answer is a purpose-based policy: define which content may train, which content may ground attributed answers, which pages should be indexed, and which data must never be public. The shared signal makes one part of that matrix cheaper to communicate; it does not decide the matrix for you.

For a site that already maintains provider controls, it is reasonable to prepare compatibility now and deploy once path-level intent is approved. Keep both layers during the transition, automate testing of the served file, and label each provider's support as announced, pilot, or active. For a site with no content-usage policy, do not start with robots.txt: first create the inventory, owners, and decisions for training, search, grounding, and access.

The October 30 development matters because OpenAI and Google backed the same interoperable building block and the IETF connected it to an explicit definition of generative training. Its real value will appear when organizations can explain a rule, prove which version they published, and reconcile it with contracts and observed behavior. If you need to turn that matrix into architecture and operable controls, Wasyra's custom agents and AI development service can help design the flow without confusing signaling, enforcement, and evidence.