# robots.txt Published articles for robots.txt. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Cloudflare update distinguishes search crawlers from AI training bots DevFeed: [Cloudflare update distinguishes search crawlers from AI training bots]() Original publisher: [Read original article]() Author: neuromamontov Published: 2026-09-16T21:09:58Z Content type: news Language: ru Sources: [Tagir Valeev]() Topics: [Cloudflare](), [Artificial Intelligence]() Tags: [2025](), [ai](), [ai-5ebc2dd0d80c](), [cloudflare](), [geo](), [llms-txt](), [robots-txt](), [tag-3ec72a25b0de](), [tag-6dbce23d3386](), [tag-ba47f5ee56ab](), [tag-c94f063f9c70](), [tag-d5d895f883db]() ### AI overview The article discusses Cloudflare's update for distinguishing search crawlers from AI training bots. It explains why website owners may want to allow search indexing while restricting AI training access, and notes that the earlier llms.txt approach was not widely adopted. ### Source excerpt Буду первым, кто не просто перепостит новость про обновление Cloudflare, а ещё и по существу прокомментирует, почему тема разделения ботов на поисковых и обучающих так важна для владельцев сайтов. Что показательно, тема ещё и совсем не новая: эти разговоры я слышал ещё года полтора назад -- тогда все обсуждали формат llms.txt, который, к слову, так толком и не взлетел. Но ключевая идея была очевидна уже тогда: желающих закрыть свой сайт от обучающих ботов в разы больше чем желающих закрыть сайт от поисковых ботов -- 17% против менее чем 1% (оценка самого Cloudflare, 2025). Читать далее ## Cloudflare Adds Setting to Block AI Training Crawlers While Allowing Search Crawlers DevFeed: [Cloudflare Adds Setting to Block AI Training Crawlers While Allowing Search Crawlers]() Original publisher: [Read original article]() Author: Alex Harper Published: 2026-09-16T17:18:57Z Content type: news Language: en Sources: [Web Designer Depot]() Topics: [Cloudflare](), [AI Bots](), [Crawler](), [Google Search](), [Artificial Intelligence]() Tags: [ai](), [ai-crawlers](), [ai-tech](), [ai-training](), [anthropic](), [artificial-intelligence](), [cloudflare](), [content-protection](), [future-of-the-web](), [google-extended](), [google-search](), [googlebot](), [openai](), [publishers](), [robots-txt](), [search](), [search-engines](), [web](), [web-design](), [web-development](), [web-publishing](), [web-scraping](), [website-traffic]() ### AI overview Cloudflare launched a Disallow AI Training setting that lets website owners allow traditional search crawlers while blocking training-only crawlers from companies including Amazon, Anthropic, Meta, and OpenAI. The article notes that robots.txt depends on crawler compliance and that blocking Google-Extended does not remove content from Google Search features such as AI Overviews or AI Mode. ### Source excerpt Cloudflare just gave website owners a new weapon against AI crawlers: keep the search traffic, block the AI training. After years of watching bots consume the web's content, publishers finally have an easier way to tell AI companies where to go. ## Your robots.txt Says Yes. Your Firewall Says 403. DevFeed: [Your robots.txt Says Yes. Your Firewall Says 403.]() Original publisher: [Read original article]() Author: Brent Leekley Published: 2026-06-11T00:00:00Z Content type: article Language: en Sources: [brent.leekley.me blog]() Topics: [Cloudflare](), [AI Bots](), [AI search](), [Crawler](), [Firewall]() Tags: [403](), [aeo](), [ai-bots](), [ai-crawl-control](), [ai-crawlers](), [ai-search](), [ai-visibility](), [blocking](), [bot-management](), [chatgpt-user](), [cloudflare](), [firewall](), [gptbot](), [oai-searchbot](), [perplexity](), [robots](), [robots-txt]() ### AI overview A field note explains how a Cloudflare AI-bot blocking setting returned 403 responses to all AI agents even though the site's robots.txt allowed AI search and blocked training crawlers. It distinguishes training crawlers, search indexers, and user-triggered fetchers, and recommends auditing enforcement at the firewall layer. ### Source excerpt A client's robots.txt welcomed AI search and blocked training crawlers, but Cloudflare's blunt AI-bot toggle was returning 403 to every AI agent at the edge. How the block was found, the AI Crawl Control fix, and why you should audit enforcement, not intent. ## Three Things Made My Blog Agent-Ready. Five I Skipped on Purpose. DevFeed: [Three Things Made My Blog Agent-Ready. Five I Skipped on Purpose.]() Original publisher: [Read original article]() Published: 2026-04-30T00:00:00Z Content type: opinion Language: en Sources: [Alex Korbonits]() Topics: [Web](), [Netlify](), [Cloudflare](), [RSS Feed]() Tags: [ai-agents](), [blog](), [cloudflare](), [ietf](), [markdown](), [mcp-oauth](), [netlify](), [robots-txt](), [rss](), [web]() ### AI overview The author describes improving a personal blog's score on Cloudflare's agent-readiness checker from 25 to 50 by adding content signals in robots.txt and Link headers through Netlify. The article argues that personal blogs should implement applicable agent-facing features while skipping API, OAuth, and MCP requirements that do not fit their purpose. ### Source excerpt Cloudflare's isitagentready.com scored korbonits.com at 25 / Level 1 'Basic Web Presence.' Two hours and three small additions later it scored 50 / Level 4 'Agent-Integrated.' The other half of the points came from checks that don't apply to a personal blog -- and shipping fake compliance for them would be theatre, not value. ## Preventing Google from Indexing Staging Sites DevFeed: [Preventing Google from Indexing Staging Sites]() Original publisher: [Read original article]() Author: andrew@nystudio107.com (Andrew Welch) Published: 2017-02-02T09:00:00Z Content type: tutorial Language: en Sources: [nystudio107 | Articles on modern web development.]() Topics: [Search engine optimization (SEO)](), [Crawler](), [Content Management System](), [Web Development]() Tags: [config](), [diluting](), [environments](), [google](), [indexing](), [insights](), [multi-environment](), [prevent](), [robots-txt](), [seo](), [seomatic](), [sites](), [staging](), [value]() ### AI overview This tutorial explains how to prevent Google and other search engines from indexing staging sites. It presents robots.txt and SEOmatic as an alternative to password protection, allowing external performance and SEO testing while avoiding duplicate-content concerns. ### Source excerpt SEOmatic and a multi-environment config can prevent Google from indexing your staging sites, and diluting your SEO value ## Commenting powered by Discourse DevFeed: [Commenting powered by Discourse]() Original publisher: [Read original article]() Author: Sam Saffron Published: 2013-12-04T03:12:48Z Content type: opinion Language: en Sources: [Sam Saffron]() Topics: [web applications](), [Software](), [Markdown](), [notifications](), [email](), [Crawler]() Tags: [blog](), [email](), [markdown](), [notifications](), [robots-txt](), [spam]() ### AI overview The author explains why Discourse powers comments on the blog. Although using it adds friction because readers must log in on another site, the author values its support for thoughtful discussions, email follow-up, Markdown editing, moderation, and spam prevention. The author reports receiving no spam over the previous 55 days. ### Source excerpt Leaving comments on this blog requires a certain amount of commitment. You have to jump to another site to log in. ac12507b13e4685b.png765x428 13.4 KB Compare this to the "least amount of friction possible". A lot of work. When I made the move to Discourse I thought of this state-of-affairs as a temporary situation. I would add the "traditional" comment box at the bottom and make it super easy to add comments, I would transparently create accounts and all that jazz. A couple of months in, I am not so sure. When I think about comments on my blog these are my priorities. Give users room to type in interesting and insightful comments. Provide great support for followup (reply by email, email notifications) Rich markdown editor with edit and preview support. Comment format must be markdown, anything else and I am risking complex conversion later on. Zero spam Comments / emails / content are unconditionally hosted on my server and under my control. Not hosted by some third party under their rules with their advertising injected and my readers tracked. Trivial for me to moderate. You may notice that, "Make it super easy for anybody on the Internet to contribute a random unfiltered opinion" is surprisingly missing from this list. What does Discourse score in my 7 point dream list? A solid 7 out of 7. The extra friction completely eradicated spam and enables me to have rich conversations with my readers. I have their emails, I can communicate with them. ###About Spam twitter.com Kelly Sommers @kellabyte Trying out disqus cuz I just got 600 more spam comments on my blog since earlier. Gotta end this madness. 12:19 AM - 1 Dec 2013 1 I eliminated the vast majority of spam on this blog a while back. I blogged about it 2 years ago. Critics claimed that this approach was doomed to fail if it ever got popular. Discourse is popular. Yet I get zero spam. By zero I mean that in the last 55 days I got no spam on this blog, nor did I have to delete any spam from this blog. http://discu