# OAI-SearchBot

Published articles for OAI-SearchBot.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Your robots.txt Says Yes. Your Firewall Says 403.

DevFeed: [Your robots.txt Says Yes. Your Firewall Says 403.](<https://devfeed.tech/articles/your-robots-txt-says-yes-your-firewall-says-403-30874.md>)

Original publisher: [Read original article](<https://brent.leekley.me/blog/robots-vs-firewall/>)

Author: Brent Leekley

Published: 2026-06-11T00:00:00Z

Content type: article

Language: en

Sources: [Brent Leekley](<https://devfeed.tech/sources/brent-leekley-me-blog.md>)

Topics: [Cloudflare](<https://devfeed.tech/topics/cloudflare.md>), [AI Bots](<https://devfeed.tech/topics/ai-bots.md>), [AI search](<https://devfeed.tech/topics/ai-search.md>), [Crawler](<https://devfeed.tech/topics/crawler.md>), [Firewall](<https://devfeed.tech/topics/firewall.md>)

Tags: [403](<https://devfeed.tech/tags/403.md>), [aeo](<https://devfeed.tech/tags/aeo.md>), [ai-bots](<https://devfeed.tech/tags/ai-bots.md>), [ai-crawl-control](<https://devfeed.tech/tags/ai-crawl-control.md>), [ai-crawlers](<https://devfeed.tech/tags/ai-crawlers.md>), [ai-search](<https://devfeed.tech/tags/ai-search.md>), [ai-visibility](<https://devfeed.tech/tags/ai-visibility.md>), [blocking](<https://devfeed.tech/tags/blocking.md>), [bot-management](<https://devfeed.tech/tags/bot-management.md>), [chatgpt-user](<https://devfeed.tech/tags/chatgpt-user.md>), [cloudflare](<https://devfeed.tech/tags/cloudflare.md>), [firewall](<https://devfeed.tech/tags/firewall.md>), [gptbot](<https://devfeed.tech/tags/gptbot.md>), [oai-searchbot](<https://devfeed.tech/tags/oai-searchbot.md>), [perplexity](<https://devfeed.tech/tags/perplexity.md>), [robots](<https://devfeed.tech/tags/robots.md>), [robots-txt](<https://devfeed.tech/tags/robots-txt.md>)

### AI overview

A field note explains how a Cloudflare AI-bot blocking setting returned 403 responses to all AI agents even though the site's robots.txt allowed AI search and blocked training crawlers. It distinguishes training crawlers, search indexers, and user-triggered fetchers, and recommends auditing enforcement at the firewall layer.

### Source excerpt

A client's robots.txt welcomed AI search and blocked training crawlers, but Cloudflare's blunt AI-bot toggle was returning 403 to every AI agent at the edge. How the block was found, the AI Crawl Control fix, and why you should audit enforcement, not intent.

## robots.txt for AI crawlers

DevFeed: [robots.txt for AI crawlers](<https://devfeed.tech/articles/robots-txt-for-ai-crawlers-89087.md>)

Original publisher: [Read original article](<https://specification.website/spec/agent-readiness/robots-for-ai-crawlers/>)

Author: Joost de Valk

Published: 2026-05-29T12:14:17Z

Content type: tutorial

Language: en

Sources: [The Website Specification](<https://devfeed.tech/sources/the-website-specification.md>)

Topics: [AI Bots](<https://devfeed.tech/topics/ai-bots.md>), [Web Scraping](<https://devfeed.tech/topics/web-scraping.md>)

Tags: [agent-readiness](<https://devfeed.tech/tags/agent-readiness.md>), [ai-crawlers](<https://devfeed.tech/tags/ai-crawlers.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [blocking](<https://devfeed.tech/tags/blocking.md>), [bytedance](<https://devfeed.tech/tags/bytedance.md>), [chatgpt-user](<https://devfeed.tech/tags/chatgpt-user.md>), [claudebot](<https://devfeed.tech/tags/claudebot.md>), [crawlers](<https://devfeed.tech/tags/crawlers.md>), [gemini](<https://devfeed.tech/tags/gemini.md>), [google](<https://devfeed.tech/tags/google.md>), [google-extended](<https://devfeed.tech/tags/google-extended.md>), [gptbot](<https://devfeed.tech/tags/gptbot.md>), [https](<https://devfeed.tech/tags/https.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [llms](<https://devfeed.tech/tags/llms.md>), [oai-searchbot](<https://devfeed.tech/tags/oai-searchbot.md>), [openai](<https://devfeed.tech/tags/openai.md>), [perplexity](<https://devfeed.tech/tags/perplexity.md>), [recommended](<https://devfeed.tech/tags/recommended.md>), [robots-txt](<https://devfeed.tech/tags/robots-txt.md>)

### AI overview

The article explains how site operators can use robots.txt rules to manage AI crawlers individually, allowing retrieval bots while blocking training crawlers, or choosing the reverse. It outlines crawler identities, example rules, limitations of robots.txt, and ways to verify configuration.

### Source excerpt

Major AI vendors publish named user-agents for their crawlers. Setting an explicit allow or disallow per agent is the clearest way to control how your content is used.