# AI crawlers Published articles for AI crawlers. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Cloudflare Adds Setting to Block AI Training Crawlers While Allowing Search Crawlers DevFeed: [Cloudflare Adds Setting to Block AI Training Crawlers While Allowing Search Crawlers]() Original publisher: [Read original article]() Author: Alex Harper Published: 2026-09-16T17:18:57Z Content type: news Language: en Sources: [Web Designer Depot]() Topics: [Cloudflare](), [AI Bots](), [Crawler](), [Google Search](), [Artificial Intelligence]() Tags: [ai](), [ai-crawlers](), [ai-tech](), [ai-training](), [anthropic](), [artificial-intelligence](), [cloudflare](), [content-protection](), [future-of-the-web](), [google-extended](), [google-search](), [googlebot](), [openai](), [publishers](), [robots-txt](), [search](), [search-engines](), [web](), [web-design](), [web-development](), [web-publishing](), [web-scraping](), [website-traffic]() ### AI overview Cloudflare launched a Disallow AI Training setting that lets website owners allow traditional search crawlers while blocking training-only crawlers from companies including Amazon, Anthropic, Meta, and OpenAI. The article notes that robots.txt depends on crawler compliance and that blocking Google-Extended does not remove content from Google Search features such as AI Overviews or AI Mode. ### Source excerpt Cloudflare just gave website owners a new weapon against AI crawlers: keep the search traffic, block the AI training. After years of watching bots consume the web's content, publishers finally have an easier way to tell AI companies where to go. ## Keeping AI crawlers off my Forgejo server DevFeed: [Keeping AI crawlers off my Forgejo server]() Original publisher: [Read original article]() Author: Harsh Shandilya Published: 2026-06-29T04:53:18Z Content type: article Language: en Sources: [Posts on Harsh Shandilya]() Topics: [forgejo](), [AI Bots](), [Cloudflare](), [Security](), [Fail2ban](), [Server]() Tags: [ai](), [ai-crawlers](), [cloudflare](), [crawlers](), [fail2ban](), [forgejo](), [security](), [server]() ### AI overview A developer describes mitigating AI crawler traffic against a Forgejo instance using fail2ban, Caddy configuration, and Cloudflare rules. The measures reduced some traffic but also exposed client-IP handling and availability issues, while IP bans approached Cloudflare's access-rule limit. ### Source excerpt The short and bumbling journey to finally giving my tiny VPS some respite ## Your robots.txt Says Yes. Your Firewall Says 403. DevFeed: [Your robots.txt Says Yes. Your Firewall Says 403.]() Original publisher: [Read original article]() Author: Brent Leekley Published: 2026-06-11T00:00:00Z Content type: article Language: en Sources: [brent.leekley.me blog]() Topics: [Cloudflare](), [AI Bots](), [AI search](), [Crawler](), [Firewall]() Tags: [403](), [aeo](), [ai-bots](), [ai-crawl-control](), [ai-crawlers](), [ai-search](), [ai-visibility](), [blocking](), [bot-management](), [chatgpt-user](), [cloudflare](), [firewall](), [gptbot](), [oai-searchbot](), [perplexity](), [robots](), [robots-txt]() ### AI overview A field note explains how a Cloudflare AI-bot blocking setting returned 403 responses to all AI agents even though the site's robots.txt allowed AI search and blocked training crawlers. It distinguishes training crawlers, search indexers, and user-triggered fetchers, and recommends auditing enforcement at the firewall layer. ### Source excerpt A client's robots.txt welcomed AI search and blocked training crawlers, but Cloudflare's blunt AI-bot toggle was returning 403 to every AI agent at the edge. How the block was found, the AI Crawl Control fix, and why you should audit enforcement, not intent. ## How to add llms.txt to a Hugo Blog DevFeed: [How to add llms.txt to a Hugo Blog]() Original publisher: [Read original article]() Author: Tom Published: 2025-10-16T00:00:00Z Content type: tutorial Language: en Sources: [Tomasz Gągor]() Topics: [Hugo](), [AI Bots](), [Search engine optimization (SEO)]() Tags: [ai](), [ai-content-visibility](), [ai-crawlers](), [blog](), [generative-engine-optimization](), [geo](), [how-to](), [hugo](), [hugo-llms-txt](), [hugo-seo](), [llm-metadata](), [markdown](), [seo]() ### AI overview A tutorial showing how to add an llms.txt Markdown file to a Hugo blog using a custom media type, output format, and template. It presents the file as a way to help AI systems understand and discover site content, while noting that its effect on organic traffic is uncertain. ### Source excerpt Learn how to add an llms.txt file to your Hugo blog to make it more visible to AI agents and improve Generative Engine Optimization (GEO). ## What is llms.txt? Breaking down the skepticism DevFeed: [What is llms.txt? Breaking down the skepticism]() Original publisher: [Read original article]() Author: Tiffany Chen Published: 2025-04-02T00:00:00Z Content type: article Language: en Sources: [Mintlify Blog]() Topics: [Large Language Model](), [Documentation](), [AI Bots](), [Search engine optimization (SEO)](), [Web]() Tags: [ai-crawlers](), [ai-trends](), [article](), [documentation](), [javascript](), [llms](), [seo]() ### AI overview This article explains llms.txt, a proposed Markdown standard that provides large language models and AI crawlers with a structured summary and reading guide for website documentation. It describes early adoption, ongoing skepticism, and the distinction between improving AI context and improving search-engine traffic. ### Source excerpt You might be seeing llms.txt pop up more lately. It's a new standard that's quickly gaining traction that makes web content easier for large language models (LLMs) to understand.