# Bot Management Published articles for Bot Management. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Cloudflare adds controls to allow search indexing while disallowing AI training DevFeed: [Cloudflare adds controls to allow search indexing while disallowing AI training]() Original publisher: [Read original article]() Author: Bryan Becker Published: 2026-09-15T13:00:00Z Content type: release Language: en Sources: [Cloudflare Blog]() Topics: [Cloudflare](), [Artificial Intelligence](), [Crawler](), [Google](), [Microsoft]() Tags: [ai](), [ai-bots](), [blocking](), [bot-management](), [bots](), [cloudflare](), [content](), [google](), [internet](), [microsoft](), [network-services](), [product-news](), [search](), [security]() ### AI overview Cloudflare announces a Disallow AI Training setting that lets website owners remain indexed in search while refusing AI training by mixed-use crawlers. Apple, Google, and Microsoft honor or have committed to honor the setting. ### Source excerpt Cloudflare is giving site owners a way to stay discoverable while disallowing AI training. New controls and an Accountable designation establish a shared model with Apple, Google, and Microsoft. ## Introducing Adaptive Intelligence: Undermining the economics of every bot attack DevFeed: [Introducing Adaptive Intelligence: Undermining the economics of every bot attack]() Original publisher: [Read original article]() Author: Chris Pope Published: 2026-08-31T12:59:00Z Content type: release Language: en Sources: [Cloudflare Blog]() Topics: [Bot](), [Artificial Intelligence](), [Cloudflare](), [Network](), [releases]() Tags: [ai](), [application-security](), [attacks](), [bot-management](), [cloudflare](), [machine-learning](), [network](), [product-news](), [releases]() ### AI overview Cloudflare introduces Adaptive Intelligence, a bot detection engine designed to undermine the economics of automated attacks. It responds to attackers' changing tactics by learning from live traffic and making attacks slower and more costly to sustain. ### Source excerpt Bot operators have historically had the economic advantage, bypassing static, deterministic detection rules with cheap proxies and retooling. Cloudflare's new Adaptive Intelligence engine flips this dynamic by autonomously learning from the meta-signals of live traffic and deploying disposable rules, making automated attacks too expensive to sustain. ## BotBase for Operators: A clearer path to joining Cloudflare's directory of bots and agents DevFeed: [BotBase for Operators: A clearer path to joining Cloudflare's directory of bots and agents]() Original publisher: [Read original article]() Author: Julian Laxman Published: 2026-08-28T12:59:44Z Content type: release Language: en Sources: [Cloudflare Blog]() Topics: [AI Bots]() Tags: [agents](), [ai-bots](), [application-security](), [bot-management](), [bots](), [cloudflare](), [content](), [model](), [update]() ### AI overview Cloudflare launched BotBase for Operators, adding a dashboard area where bot operators can browse the directory, submit bots, and track submission history. The update also provides submission-status visibility, editing, and a behavior model for declaring content use. ### Source excerpt Bot operators now have a home in the Cloudflare dashboard to manage submissions. This update adds submission status tracking, submission editing, and a behavior model so operators can accurately declare how their bots use content. ## Say it once: Introducing Bot Preference Sync DevFeed: [Say it once: Introducing Bot Preference Sync]() Original publisher: [Read original article]() Author: Jin-Hee Lee Published: 2026-08-21T23:19:57Z Content type: article Language: en Sources: [Cloudflare Blog]() Topics: [AI Bots]() Tags: [ai](), [ai-bots](), [ai-models](), [ai-training](), [bot-management](), [bots](), [cloudflare](), [network-services](), [product-news](), [robots](), [search](), [sync](), [training]() ### AI overview Cloudflare introduces Bot Preference Sync, which updates robots.txt preferences to match configured AI bot policies for Search, Agent, and Training traffic. ### Source excerpt Cloudflare's new Bot Preference Sync automatically aligns your robots.txt file with your AI bot policies for Search, Agent, and Training. Easily manage which bots access your content without maintaining static files. ## Your robots.txt Says Yes. Your Firewall Says 403. DevFeed: [Your robots.txt Says Yes. Your Firewall Says 403.]() Original publisher: [Read original article]() Author: Brent Leekley Published: 2026-06-11T00:00:00Z Content type: article Language: en Sources: [brent.leekley.me blog]() Topics: [Cloudflare](), [AI Bots](), [AI search](), [Crawler](), [Firewall]() Tags: [403](), [aeo](), [ai-bots](), [ai-crawl-control](), [ai-crawlers](), [ai-search](), [ai-visibility](), [blocking](), [bot-management](), [chatgpt-user](), [cloudflare](), [firewall](), [gptbot](), [oai-searchbot](), [perplexity](), [robots](), [robots-txt]() ### AI overview A field note explains how a Cloudflare AI-bot blocking setting returned 403 responses to all AI agents even though the site's robots.txt allowed AI search and blocked training crawlers. It distinguishes training crawlers, search indexers, and user-triggered fetchers, and recommends auditing enforcement at the firewall layer. ### Source excerpt A client's robots.txt welcomed AI search and blocked training crawlers, but Cloudflare's blunt AI-bot toggle was returning 403 to every AI agent at the edge. How the block was found, the AI Crawl Control fix, and why you should audit enforcement, not intent. ## Reliability lessons from the 2025 Cloudflare outage DevFeed: [Reliability lessons from the 2025 Cloudflare outage]() Original publisher: [Read original article]() Author: Andre Newman Published: 2025-11-20T00:00:00Z Content type: article Language: en Sources: [Gremlin Blog]() Topics: [Cloudflare](), [Bot Management](), [configuration](), [Chaos Engineering](), [Post Mortem](), [Network](), [Workers]() Tags: [2025](), [after-the-retrospective](), [bot-management](), [chaos-engineering](), [cloudflare](), [configuration](), [engineering](), [http](), [internet](), [management](), [outage](), [outages](), [post-mortem](), [workers]() ### AI overview The article examines the November 2025 Cloudflare outage, explaining how an oversized Bot Management configuration caused HTTP 5XX errors and cascading failures across dependent services. It highlights configuration propagation, service dependencies, and chaos engineering as reliability considerations. ### Source excerpt In November 2025, a misconfigured Cloudflare service led to a partial outage. Learn what happened, and what you can do to reduce the impact of similar outages.