# web scraping

Published articles for web scraping.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Cloudflare Adds Setting to Block AI Training Crawlers While Allowing Search Crawlers

DevFeed: [Cloudflare Adds Setting to Block AI Training Crawlers While Allowing Search Crawlers](<https://devfeed.tech/articles/cloudflare-just-gave-ai-training-bots-the-middle-finger-31387.md>)

Original publisher: [Read original article](<https://webdesignerdepot.com/cloudflare-just-gave-ai-training-bots-the-middle-finger/>)

Author: Alex Harper

Published: 2026-09-16T17:18:57Z

Content type: news

Language: en

Sources: [Web Designer Depot](<https://devfeed.tech/sources/web-designer-depot.md>)

Topics: [Cloudflare](<https://devfeed.tech/topics/cloudflare.md>), [AI Bots](<https://devfeed.tech/topics/ai-bots.md>), [Crawler](<https://devfeed.tech/topics/crawler.md>), [Google Search](<https://devfeed.tech/topics/google-search.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-crawlers](<https://devfeed.tech/tags/ai-crawlers.md>), [ai-tech](<https://devfeed.tech/tags/ai-tech.md>), [ai-training](<https://devfeed.tech/tags/ai-training.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [cloudflare](<https://devfeed.tech/tags/cloudflare.md>), [content-protection](<https://devfeed.tech/tags/content-protection.md>), [future-of-the-web](<https://devfeed.tech/tags/future-of-the-web.md>), [google-extended](<https://devfeed.tech/tags/google-extended.md>), [google-search](<https://devfeed.tech/tags/google-search.md>), [googlebot](<https://devfeed.tech/tags/googlebot.md>), [openai](<https://devfeed.tech/tags/openai.md>), [publishers](<https://devfeed.tech/tags/publishers.md>), [robots-txt](<https://devfeed.tech/tags/robots-txt.md>), [search](<https://devfeed.tech/tags/search.md>), [search-engines](<https://devfeed.tech/tags/search-engines.md>), [web](<https://devfeed.tech/tags/web.md>), [web-design](<https://devfeed.tech/tags/web-design.md>), [web-development](<https://devfeed.tech/tags/web-development.md>), [web-publishing](<https://devfeed.tech/tags/web-publishing.md>), [web-scraping](<https://devfeed.tech/tags/web-scraping.md>), [website-traffic](<https://devfeed.tech/tags/website-traffic.md>)

### AI overview

Cloudflare launched a Disallow AI Training setting that lets website owners allow traditional search crawlers while blocking training-only crawlers from companies including Amazon, Anthropic, Meta, and OpenAI. The article notes that robots.txt depends on crawler compliance and that blocking Google-Extended does not remove content from Google Search features such as AI Overviews or AI Mode.

### Source excerpt

Cloudflare just gave website owners a new weapon against AI crawlers: keep the search traffic, block the AI training. After years of watching bots consume the web's content, publishers finally have an easier way to tell AI companies where to go.

## Building an AI-automated Reddit trend scraper with Honojs, OpenRouter, and Scrapefast

DevFeed: [Building an AI-automated Reddit trend scraper with Honojs, OpenRouter, and Scrapefast](<https://devfeed.tech/articles/ai-automated-reddit-trend-scraper-that-saved-us-1000-month-on-content-creation-39116.md>)

Original publisher: [Read original article](<https://ihatereading.in/t/ai-automated-reddit-trend-scraper-that-saved-us-1000-month-on-content-creation>)

Author: iHateReading

Published: 2026-08-29T02:47:02Z

Content type: tutorial

Language: en

Sources: [iHateReading](<https://devfeed.tech/sources/ihatereading.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Hono](<https://devfeed.tech/topics/honojs.md>), [Development](<https://devfeed.tech/topics/development.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-automated-reddit-trend-scraper-that-saved-us-1000-month-on-content-creation](<https://devfeed.tech/tags/ai-automated-reddit-trend-scraper-that-saved-us-1000-month-on-content-creation.md>), [api](<https://devfeed.tech/tags/api.md>), [automated](<https://devfeed.tech/tags/automated.md>), [backend](<https://devfeed.tech/tags/backend.md>), [blockchain](<https://devfeed.tech/tags/blockchain.md>), [coding](<https://devfeed.tech/tags/coding.md>), [development](<https://devfeed.tech/tags/development.md>), [frontend](<https://devfeed.tech/tags/frontend.md>), [honojs](<https://devfeed.tech/tags/honojs.md>), [ihatereading](<https://devfeed.tech/tags/ihatereading.md>), [javascript](<https://devfeed.tech/tags/javascript.md>), [node-js](<https://devfeed.tech/tags/node-js.md>), [openrouter](<https://devfeed.tech/tags/openrouter.md>), [product](<https://devfeed.tech/tags/product.md>), [programming](<https://devfeed.tech/tags/programming.md>), [react](<https://devfeed.tech/tags/react.md>), [reddit](<https://devfeed.tech/tags/reddit.md>), [reddit-scraper-api-web-scraping-ai-agent](<https://devfeed.tech/tags/reddit-scraper-api-web-scraping-ai-agent.md>), [scrapefast](<https://devfeed.tech/tags/scrapefast.md>), [scraper](<https://devfeed.tech/tags/scraper.md>), [typescript](<https://devfeed.tech/tags/typescript.md>), [web-development](<https://devfeed.tech/tags/web-development.md>), [web-scraping](<https://devfeed.tech/tags/web-scraping.md>)

### AI overview

A developer walkthrough of building an AI-automated Reddit trend scraper with Honojs, OpenRouter, and Scrapefast.

### Source excerpt

Using Honojs, OpenRouter and Scrapefast, how we built one on our own

## One honojs file for entire web scraping API

DevFeed: [One honojs file for entire web scraping API](<https://devfeed.tech/articles/one-honojs-file-for-entire-web-scraping-api-39140.md>)

Original publisher: [Read original article](<https://ihatereading.in/t/one-honojs-file-for-entire-web-scraping-api>)

Author: iHateReading

Published: 2026-04-13T16:54:35Z

Content type: tutorial

Language: en

Sources: [iHateReading](<https://devfeed.tech/sources/ihatereading.md>)

Topics: [Hono](<https://devfeed.tech/topics/honojs.md>), [Web Scraping](<https://devfeed.tech/topics/web-scraping.md>), [API](<https://devfeed.tech/topics/api.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [backend](<https://devfeed.tech/tags/backend.md>), [blockchain](<https://devfeed.tech/tags/blockchain.md>), [coding](<https://devfeed.tech/tags/coding.md>), [development](<https://devfeed.tech/tags/development.md>), [frontend](<https://devfeed.tech/tags/frontend.md>), [github-repository](<https://devfeed.tech/tags/github-repository.md>), [honojs](<https://devfeed.tech/tags/honojs.md>), [ihatereading](<https://devfeed.tech/tags/ihatereading.md>), [javascript](<https://devfeed.tech/tags/javascript.md>), [node-js](<https://devfeed.tech/tags/node-js.md>), [one-honojs-file-for-entire-web-scraping-api](<https://devfeed.tech/tags/one-honojs-file-for-entire-web-scraping-api.md>), [product](<https://devfeed.tech/tags/product.md>), [programming](<https://devfeed.tech/tags/programming.md>), [react](<https://devfeed.tech/tags/react.md>), [scraping](<https://devfeed.tech/tags/scraping.md>), [typescript](<https://devfeed.tech/tags/typescript.md>), [web-development](<https://devfeed.tech/tags/web-development.md>), [web-scraping](<https://devfeed.tech/tags/web-scraping.md>), [web-scraping-github-repository-scraping-backend-honojs](<https://devfeed.tech/tags/web-scraping-github-repository-scraping-backend-honojs.md>)

### AI overview

A tutorial about building web-scraping endpoints in a single Honojs file.

### Source excerpt

Building scrape endpoints using one honojs file

## Data Is the Bottleneck in Many AI Projects

DevFeed: [Data Is the Bottleneck in Many AI Projects](<https://devfeed.tech/articles/the-most-important-part-of-your-ai-project-isn-t-ai-32383.md>)

Original publisher: [Read original article](<https://brianjenney.substack.com/p/the-most-important-part-of-your-ai>)

Author: Brian Jenney

Published: 2026-02-07T16:25:26Z

Content type: opinion

Language: en

Sources: [Brian Jenney](<https://devfeed.tech/sources/brian-jenney.md>)

Topics: [data](<https://devfeed.tech/topics/data.md>), [Web Scraping](<https://devfeed.tech/topics/web-scraping.md>), [Structured-data](<https://devfeed.tech/topics/structured-data.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [data](<https://devfeed.tech/tags/data.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [web-scraping](<https://devfeed.tech/tags/web-scraping.md>)

### AI overview

The article argues that data collection and preparation, rather than wiring together AI components, are often the main bottlenecks in AI projects. It discusses messy real-world data, data pipelines, vector databases, and web scraping as ways to ground useful AI outputs.

### Source excerpt

This is the first Saturday I've had off in a few weeks.