# web publishing

Published articles for web publishing.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Cloudflare Adds Setting to Block AI Training Crawlers While Allowing Search Crawlers

DevFeed: [Cloudflare Adds Setting to Block AI Training Crawlers While Allowing Search Crawlers](<https://devfeed.tech/articles/cloudflare-just-gave-ai-training-bots-the-middle-finger-31387.md>)

Original publisher: [Read original article](<https://webdesignerdepot.com/cloudflare-just-gave-ai-training-bots-the-middle-finger/>)

Author: Alex Harper

Published: 2026-09-16T17:18:57Z

Content type: news

Language: en

Sources: [Web Designer Depot](<https://devfeed.tech/sources/web-designer-depot.md>)

Topics: [Cloudflare](<https://devfeed.tech/topics/cloudflare.md>), [AI Bots](<https://devfeed.tech/topics/ai-bots.md>), [Crawler](<https://devfeed.tech/topics/crawler.md>), [Google Search](<https://devfeed.tech/topics/google-search.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-crawlers](<https://devfeed.tech/tags/ai-crawlers.md>), [ai-tech](<https://devfeed.tech/tags/ai-tech.md>), [ai-training](<https://devfeed.tech/tags/ai-training.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [cloudflare](<https://devfeed.tech/tags/cloudflare.md>), [content-protection](<https://devfeed.tech/tags/content-protection.md>), [future-of-the-web](<https://devfeed.tech/tags/future-of-the-web.md>), [google-extended](<https://devfeed.tech/tags/google-extended.md>), [google-search](<https://devfeed.tech/tags/google-search.md>), [googlebot](<https://devfeed.tech/tags/googlebot.md>), [openai](<https://devfeed.tech/tags/openai.md>), [publishers](<https://devfeed.tech/tags/publishers.md>), [robots-txt](<https://devfeed.tech/tags/robots-txt.md>), [search](<https://devfeed.tech/tags/search.md>), [search-engines](<https://devfeed.tech/tags/search-engines.md>), [web](<https://devfeed.tech/tags/web.md>), [web-design](<https://devfeed.tech/tags/web-design.md>), [web-development](<https://devfeed.tech/tags/web-development.md>), [web-publishing](<https://devfeed.tech/tags/web-publishing.md>), [web-scraping](<https://devfeed.tech/tags/web-scraping.md>), [website-traffic](<https://devfeed.tech/tags/website-traffic.md>)

### AI overview

Cloudflare launched a Disallow AI Training setting that lets website owners allow traditional search crawlers while blocking training-only crawlers from companies including Amazon, Anthropic, Meta, and OpenAI. The article notes that robots.txt depends on crawler compliance and that blocking Google-Extended does not remove content from Google Search features such as AI Overviews or AI Mode.

### Source excerpt

Cloudflare just gave website owners a new weapon against AI crawlers: keep the search traffic, block the AI training. After years of watching bots consume the web's content, publishers finally have an easier way to tell AI companies where to go.

## Muse setup revised

DevFeed: [Muse setup revised](<https://devfeed.tech/articles/muse-setup-revised-34430.md>)

Original publisher: [Read original article](<https://tapoueh.org/blog/2011/07/muse-setup-revised/>)

Author: Dimitri Fontaine PostgreSQL Major Contributor; Author

Published: 2011-07-05T17:55:00Z

Content type: article

Language: en

Sources: [Dimitri Fontaine](<https://devfeed.tech/sources/dimitri-fontaine.md>)

Topics: [Emacs](<https://devfeed.tech/topics/emacs.md>), [RSS Feed](<https://devfeed.tech/topics/rss-feed.md>), [Website](<https://devfeed.tech/topics/website.md>), [Publishing](<https://devfeed.tech/topics/publishing.md>)

Tags: [emacs](<https://devfeed.tech/tags/emacs.md>), [rss](<https://devfeed.tech/tags/rss.md>), [web-publishing](<https://devfeed.tech/tags/web-publishing.md>), [website](<https://devfeed.tech/tags/website.md>)

### AI overview

The author describes a revised website setup built with Emacs Muse and custom tapoueh.el tweaks. The setup adds tagging and support for producing multiple topic-specific RSS feeds from one blog.

### Source excerpt

Most of you are probably reading my posts directly in their RSS reader tools (mine is gnus thanks to the Gwene service), so you probably missed it, but I just pushed a whole new version of my website, still using Emacs Muse as the engine. My setup is tentatively called tapoueh.el and browsable online. It consists of some tweaks on top of Muse, so that I can enjoy tags and proper rss support. By proper, I mean that I want to be able to produce as many topic RSS feeds from a single blog, and thanks to the tags support that's now what I have.