- The documented change
- Keep the dates visible
The documented change
Cloudflare’s July 1 documentation describes updated defaults taking effect September 15 for new domains: Training and Agent bots are blocked on pages displaying ads, while Search remains allowed. It also says mixed-purpose crawlers combining search and training can be blocked by configurations that block AI training.
A scoped review, not a blanket instruction
A site owner should inspect the policy actually saved for the relevant zone and compare it with the intended treatment of each use. A general statement that search is allowed does not settle how a mixed-purpose crawler is treated.
Keep the dates visible
This is an upcoming-deadline briefing based on existing documentation, not a newly announced September 10 policy. TLB has not changed any customer’s crawler settings. Verify the provider’s current documentation and a controlled crawl test before changing a production configuration.
Read beyond this page.
Recorded source-check date: 10 Sep 2026. A link is not, by itself, evidence that every claim has been independently verified.
Changes & version history
Version 3 · 10 Sep 2026
Scheduled release of checksum-bound AI-assisted editorial review
Version 2 · 10 Sep 2026
Checksum-bound editorial review scheduled for release
Version 1 · 10 Sep 2026
Source-linked private review edition