# AI Crawl Control

Published articles for AI Crawl Control.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Your robots.txt Says Yes. Your Firewall Says 403.

DevFeed: [Your robots.txt Says Yes. Your Firewall Says 403.](<https://devfeed.tech/articles/your-robots-txt-says-yes-your-firewall-says-403-30874.md>)

Original publisher: [Read original article](<https://brent.leekley.me/blog/robots-vs-firewall/>)

Author: Brent Leekley

Published: 2026-06-11T00:00:00Z

Content type: article

Language: en

Sources: [brent.leekley.me blog](<https://devfeed.tech/sources/brent-leekley-me-blog.md>)

Topics: [Cloudflare](<https://devfeed.tech/topics/cloudflare.md>), [AI Bots](<https://devfeed.tech/topics/ai-bots.md>), [AI search](<https://devfeed.tech/topics/ai-search.md>), [Crawler](<https://devfeed.tech/topics/crawler.md>), [Firewall](<https://devfeed.tech/topics/firewall.md>)

Tags: [403](<https://devfeed.tech/tags/403.md>), [aeo](<https://devfeed.tech/tags/aeo.md>), [ai-bots](<https://devfeed.tech/tags/ai-bots.md>), [ai-crawl-control](<https://devfeed.tech/tags/ai-crawl-control.md>), [ai-crawlers](<https://devfeed.tech/tags/ai-crawlers.md>), [ai-search](<https://devfeed.tech/tags/ai-search.md>), [ai-visibility](<https://devfeed.tech/tags/ai-visibility.md>), [blocking](<https://devfeed.tech/tags/blocking.md>), [bot-management](<https://devfeed.tech/tags/bot-management.md>), [chatgpt-user](<https://devfeed.tech/tags/chatgpt-user.md>), [cloudflare](<https://devfeed.tech/tags/cloudflare.md>), [firewall](<https://devfeed.tech/tags/firewall.md>), [gptbot](<https://devfeed.tech/tags/gptbot.md>), [oai-searchbot](<https://devfeed.tech/tags/oai-searchbot.md>), [perplexity](<https://devfeed.tech/tags/perplexity.md>), [robots](<https://devfeed.tech/tags/robots.md>), [robots-txt](<https://devfeed.tech/tags/robots-txt.md>)

### AI overview

A field note explains how a Cloudflare AI-bot blocking setting returned 403 responses to all AI agents even though the site's robots.txt allowed AI search and blocked training crawlers. It distinguishes training crawlers, search indexers, and user-triggered fetchers, and recommends auditing enforcement at the firewall layer.

### Source excerpt

A client's robots.txt welcomed AI search and blocked training crawlers, but Cloudflare's blunt AI-bot toggle was returning 403 to every AI agent at the edge. How the block was found, the AI Crawl Control fix, and why you should audit enforcement, not intent.