The biggest names in AI are backing a new way to opt out of scraping
The Verge AI · 31 Oct 2026, 06:26 AEST
OpenAI, Google, Microsoft, and Anthropic are supporting a new standard for publishers to opt out of AI scraping. This initiative allows publishers to manage their AI training preferences across multiple companies more efficiently.
Key facts
- OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable.
- With this control, publishers can apply their AI training preferences across multiple AI companies, instead of setting them individually.
- The shared 'policy token' signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF).
- It will live within a website's robots.txt file, which has long signaled which web crawlers can and can't scrape a site.
Read aloud
Open A I, Google, Microsoft, and Anthropic are supporting a new standard for publishers to opt out of A I scraping. This initiative allows publishers to manage their A I training preferences across multiple companies more efficiently. Open A I, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by A I, as reported earlier by Search Engine Roundtable. With this control, publishers can apply their A I training preferences across multiple A I companies, instead of setting them individually. The shared 'policy token' signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force IETF. It will live within a website's robots.txt file, which has long signaled which web crawlers can and can't scrape a site.
Sources in this cluster
- The biggest names in AI are backing a new way to opt out of scraping
The Verge AI · 31 Oct 2026, 06:26 AEST