# Crawler An automated web client that can recursively traverse links for indexing. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Cloudflare Adds Setting to Block AI Training Crawlers While Allowing Search Crawlers DevFeed: [Cloudflare Adds Setting to Block AI Training Crawlers While Allowing Search Crawlers]() Original publisher: [Read original article]() Author: Alex Harper Published: 2026-09-16T17:18:57Z Content type: news Language: en Sources: [Web Designer Depot]() Topics: [Cloudflare](), [AI Bots](), [Crawler](), [Google Search](), [Artificial Intelligence]() Tags: [ai](), [ai-crawlers](), [ai-tech](), [ai-training](), [anthropic](), [artificial-intelligence](), [cloudflare](), [content-protection](), [future-of-the-web](), [google-extended](), [google-search](), [googlebot](), [openai](), [publishers](), [robots-txt](), [search](), [search-engines](), [web](), [web-design](), [web-development](), [web-publishing](), [web-scraping](), [website-traffic]() ### AI overview Cloudflare launched a Disallow AI Training setting that lets website owners allow traditional search crawlers while blocking training-only crawlers from companies including Amazon, Anthropic, Meta, and OpenAI. The article notes that robots.txt depends on crawler compliance and that blocking Google-Extended does not remove content from Google Search features such as AI Overviews or AI Mode. ### Source excerpt Cloudflare just gave website owners a new weapon against AI crawlers: keep the search traffic, block the AI training. After years of watching bots consume the web's content, publishers finally have an easier way to tell AI companies where to go. ## Cloudflare adds controls to allow search indexing while disallowing AI training DevFeed: [Cloudflare adds controls to allow search indexing while disallowing AI training]() Original publisher: [Read original article]() Author: Bryan Becker Published: 2026-09-15T13:00:00Z Content type: release Language: en Sources: [Cloudflare Blog]() Topics: [Cloudflare](), [Artificial Intelligence](), [Crawler](), [Google](), [Microsoft]() Tags: [ai](), [ai-bots](), [blocking](), [bot-management](), [bots](), [cloudflare](), [content](), [google](), [internet](), [microsoft](), [network-services](), [product-news](), [search](), [security]() ### AI overview Cloudflare announces a Disallow AI Training setting that lets website owners remain indexed in search while refusing AI training by mixed-use crawlers. Apple, Google, and Microsoft honor or have committed to honor the setting. ### Source excerpt Cloudflare is giving site owners a way to stay discoverable while disallowing AI training. New controls and an Accountable designation establish a shared model with Apple, Google, and Microsoft. ## GHarchive data has become unreliable for measuring GitHub activity DevFeed: [GHarchive data has become unreliable for measuring GitHub activity]() Original publisher: [Read original article]() Author: KD (noreply@blogger.com) Published: 2026-09-03T16:00:00Z Content type: opinion Language: en Sources: [Google Open Source Blog]() Topics: [data](), [datasets](), [GitHub](), [Crawler](), [GitHub API](), [Open Source](), [GraphQL]() Tags: [collect](), [data](), [dataset](), [datasets](), [github](), [google](), [google-open-source](), [graphql](), [open-data-sets](), [open-source](), [oss](), [pipeline](), [pull-requests](), [retention](), [stream](), [volume]() ### AI overview This article examines the reliability of open source data, focusing on GHarchive's coverage of GitHub events. It argues that GHarchive should not be used for real-time or volume-based metrics because event retention has declined and some activity is omitted by the GitHub Event stream and API limitations. ### Source excerpt by Sophia Vargas, Google Open Source & Andrew Nesbitt, Ecosyste.ms Every second, open source contribution quietly shapes the software we rely on, and yet our view of this open ecosystem is surprisingly opaque. Open source development is performed in public spaces -- we can see the commits, issues and comments, the APIs and endpoints are free to use -- the logs are just sitting there, so why can't we just collect all of the data? ...Said every researcher, everywhere. However in most cases of open source related data, we are only looking at part of the whole. Why am I writing this post? Because many of us (including many business decision-makers) are too comfortable with unsubstantiated data. We've gotten used to it. Our models assume that it's smelly and we adjust the logic and weights to compromise. When it comes to open source, our confidence is even lower, even though our resulting decisions can directly impact individuals whom we collectively depend on. Let's consider one of my favorite datasets: GHarchive. Started as a hobby project in 2011, this crawler has amassed more than 15 years of event data from GitHub. While this source provides a historical record of open source development on GitHub, as a real-time or comprehensive source of metrics, it's unreliable and should not be a source for volume-based metrics. In 2025, GHarchive captured 14% fewer events than in 2024, despite steady growth in platform adoption. Since 2025, we estimate that data retention in GHarchive has fallen to ~50% and in 2026 it may be as low as 20% for some event types (see figure below). Prior to 2025, you could make the general assumption that the majority of events would be represented in this pipeline. Since 2025, we must now assume we may be missing at least half of events and possibly more -- not to mention all of the additional activity that's left out of the event API (see GitHub's GraphQL API.) The crawler logic behind this dataset is simple: give me all the events from the GitHub Ev ## TIME serves bots a different website, and User-Agent is now a billing identity DevFeed: [TIME serves bots a different website, and User-Agent is now a billing identity]() Original publisher: [Read original article]() Author: WorkOS Published: 2026-08-06T01:46:28Z Content type: opinion Language: en Sources: [WorkOS Blog]() Topics: [AI Bots](), [Bot](), [Crawler](), [SIEM, Security, Observability]() Tags: [ads](), [billing](), [bots](), [curl](), [logs](), [measurement](), [openai](), [routing]() ### AI overview The article examines how TIME serves different content to crawlers based on User-Agent headers and records bot reads as billable ad impressions. It argues that because clients can freely change the header, crawler identity is an unreliable basis for billing and measurement. ### Source excerpt TIME forks its content per crawler and logs each bot read as a billable ad impression. The routing key for that whole ledger is a header any client can type. ## Your robots.txt Says Yes. Your Firewall Says 403. DevFeed: [Your robots.txt Says Yes. Your Firewall Says 403.]() Original publisher: [Read original article]() Author: Brent Leekley Published: 2026-06-11T00:00:00Z Content type: article Language: en Sources: [brent.leekley.me blog]() Topics: [Cloudflare](), [AI Bots](), [AI search](), [Crawler](), [Firewall]() Tags: [403](), [aeo](), [ai-bots](), [ai-crawl-control](), [ai-crawlers](), [ai-search](), [ai-visibility](), [blocking](), [bot-management](), [chatgpt-user](), [cloudflare](), [firewall](), [gptbot](), [oai-searchbot](), [perplexity](), [robots](), [robots-txt]() ### AI overview A field note explains how a Cloudflare AI-bot blocking setting returned 403 responses to all AI agents even though the site's robots.txt allowed AI search and blocked training crawlers. It distinguishes training crawlers, search indexers, and user-triggered fetchers, and recommends auditing enforcement at the firewall layer. ### Source excerpt A client's robots.txt welcomed AI search and blocked training crawlers, but Cloudflare's blunt AI-bot toggle was returning 403 to every AI agent at the edge. How the block was found, the AI Crawl Control fix, and why you should audit enforcement, not intent. ## How often do LLMs visit llms.txt? DevFeed: [How often do LLMs visit llms.txt?]() Original publisher: [Read original article]() Author: Tiffany Chen Published: 2025-06-27T00:00:00Z Content type: article Language: en Sources: [Mintlify Blog]() Topics: [Large Language Model](), [Markdown](), [Data analysis](), [Crawler](), [Retrieval-Augmented Generation]() Tags: [ai-trends](), [claude](), [data-analysis](), [google](), [llms](), [markdown](), [pages](), [retrieval](), [retrieval-augmented-generation]() ### AI overview The article presents a Profound data analysis of how language models access llms.txt and llms-full.txt. The supplied text says both files receive AI traffic, with a strong preference for llms-full.txt and ChatGPT accounting for most visits. It attributes this preference to models embedding full content instead of relying on retrieval-augmented generation, while noting that the analysis covers 25 companies. ### Source excerpt Last month, we explored signals for the emerging standard of llms.txt, which is a Markdown file that makes websites easier for LLMs to index. ## SourceHut author describes the operational cost of aggressive LLM crawlers DevFeed: [SourceHut author describes the operational cost of aggressive LLM crawlers]() Original publisher: [Read original article]() Author: March Published: 2025-03-17T00:00:00Z Content type: opinion Language: en Sources: [Drew DeVault]() Topics: [Large Language Model](), [Crawler](), [Git](), [Go Language](), [ci](), [Cryptocurrency](), [HTTP]() Tags: [agents](), [ci](), [cryptocurrency](), [git](), [go](), [http](), [llms](), [robots]() ### AI overview A personal opinion post describes SourceHut's experience mitigating aggressive LLM crawlers that ignore robots.txt, crawl expensive Git endpoints, distribute requests across many IP addresses, and contribute to recurring outages. It also compares this burden with earlier CI cryptocurrency-mining abuse and Go module mirror traffic. ### Source excerpt This blog post is expressing personal experiences and opinions and doesn't reflect any official policies of SourceHut. Over the past few months, instead of working on our priorities at SourceHut, I have spent anywhere from 20-100% of my time in any given week mitigating hyper-aggressive LLM crawlers at scale. This isn't the first time SourceHut has been at the wrong end of some malicious bullshit or paid someone else's externalized costs - every couple of years someone invents a new way of ruining my day. Four years ago, we decided to require payment to use our CI services because it was being abused to mine cryptocurrency. We alternated between periods of designing and deploying tools to curb this abuse and periods of near-complete outage when they adapted to our mitigations and saturated all of our compute with miners seeking a profit. It was bad enough having to beg my friends and family to avoid "investing" in the scam without having the scam break into my business and trash the place every day. Two years ago, we threatened to blacklist the Go module mirror because for some reason the Go team thinks that running terabytes of git clones all day, every day for every Go project on git.sr.ht is cheaper than maintaining any state or using webhooks or coordinating the work between instances or even just designing a module system that doesn't require Google to DoS git forges whose entire annual budgets are considerably smaller than a single Google engineer's salary. Now it's LLMs. If you think these crawlers respect robots.txt then you are several assumptions of good faith removed from reality. These bots crawl everything they can find, robots.txt be damned, including expensive endpoints like git blame, every page of every git log, and every commit in every repo, and they do so using random User-Agents that overlap with end-users and come from tens of thousands of IP addresses - mostly residential, in unrelated subnets, each one making no more than one HTTP request ove ## Exploring the intertwingularity of a docs site DevFeed: [Exploring the intertwingularity of a docs site]() Original publisher: [Read original article]() Published: 2024-10-16T00:00:00Z Content type: article Language: en Sources: [technicalwriting.dev]() Topics: [Crawler](), [Documentation](), [Website](), [Web]() Tags: [building](), [docs](), [exploring](), [intertwingularity](), [links](), [web]() ### AI overview The article describes building a web crawler to measure how pages in a documentation site link to one another and to the wider web. The backlink data is intended to help identify important pages and prioritize technical-writing effort. ### Source excerpt Building a web crawler to quantify interconnected knowledge. ## SEO Myths: Top 5 Sitemap Myths Demystified DevFeed: [SEO Myths: Top 5 Sitemap Myths Demystified]() Original publisher: [Read original article]() Author: andrew@nystudio107.com (Andrew Welch) Published: 2023-10-30T20:03:00Z Content type: article Language: en Sources: [nystudio107 | Articles on modern web development.]() Topics: [Search engine optimization (SEO)](), [Google Search](), [Crawler]() Tags: [2005](), [article](), [demystify](), [google-search](), [insights](), [misconceptions](), [myths](), [seo](), [sitemaps](), [surrounding](), [we-ll]() ### AI overview This article examines five common myths about sitemaps and explains that they are optional signals search engines may use, not directives. It also describes situations where a sitemap may help, such as large or new sites and sites with rich media. ### Source excerpt Sitemaps have been around since 2005, but there are still many myths and misconceptions surrounding them. We'll demystify five SEO myths in this article. ## DocSearch migration DevFeed: [DocSearch migration]() Original publisher: [Read original article]() Author: Sébastien Lorber Published: 2021-11-21T00:00:00Z Content type: article Language: en Sources: [Docusaurus Blog]() Topics: [Algolia](), [migration](), [configuration](), [Crawler](), [account]() Tags: [account](), [algolia](), [configuration](), [crawler](), [documentation](), [github](), [guide](), [migration](), [open-source](), [search]() ### AI overview DocSearch is migrating to a new system that gives users their own Algolia application and new credentials. Docusaurus site owners must update their configuration by February 1, 2022, after which existing search indexes will become read-only. ### Source excerpt DocSearch is migrating to a new, more powerful system, which gives users their own Algolia application and new credentials. ## Going Full Static DevFeed: [Going Full Static]() Original publisher: [Read original article]() Published: 2020-06-18T00:00:00Z Content type: release Language: en Sources: [The Nuxt Blog]() Topics: [Nuxt.js](), [Jamstack](), [Server-side rendering](), [Crawler](), [HTML](), [Single-page application (SPA)](), [Web Development]() Tags: [hosting](), [html](), [jamstack](), [navigation](), [nuxt](), [performance](), [release](), [server-side-rendering](), [ssr]() ### AI overview Nuxt 2.13 introduces full static export for JAMstack applications. The static target pre-renders pages to HTML, extracts payloads for client-side navigation without API calls, improves smart prefetching, includes a crawler, supports SPA fallback behavior, and improves the development experience and deployment speed. ### Source excerpt Long awaited features for JAMstack fans has been shipped in v2.13: full static export, improved smart prefetching, integrated crawler, faster re-deploy, built-in web server and new target option for config ⚡ ## Building a legacy search engine for a legacy protocol DevFeed: [Building a legacy search engine for a legacy protocol]() Original publisher: [Read original article]() Author: ben@benjojo.co.uk Published: 2017-05-21T12:26:48Z Content type: article Language: en Sources: [benjojo blog]() Topics: [Protocol (disambiguation)](), [Crawler](), [search-engines](), [legacy](), [selectors](), [Windows](), [Bash](), [win32](), [Support]() Tags: [bash](), [crawler](), [css](), [http](), [implementation](), [legacy](), [protocol](), [search](), [search-engine](), [server](), [servers](), [software](), [tcp]() ### AI overview The article describes building a search engine for the Gopher protocol. It covers crawling Gopher servers, indexing menus and text files, and running an old AltaVista search engine on Windows 98 with a stunnel relay. ### Source excerpt Building a legacy search engine for a legacy protocol Translations are availa ## Preventing Google from Indexing Staging Sites DevFeed: [Preventing Google from Indexing Staging Sites]() Original publisher: [Read original article]() Author: andrew@nystudio107.com (Andrew Welch) Published: 2017-02-02T09:00:00Z Content type: tutorial Language: en Sources: [nystudio107 | Articles on modern web development.]() Topics: [Search engine optimization (SEO)](), [Crawler](), [Content Management System](), [Web Development]() Tags: [config](), [diluting](), [environments](), [google](), [indexing](), [insights](), [multi-environment](), [prevent](), [robots-txt](), [seo](), [seomatic](), [sites](), [staging](), [value]() ### AI overview This tutorial explains how to prevent Google and other search engines from indexing staging sites. It presents robots.txt and SEOmatic as an alternative to password protection, allowing external performance and SEO testing while avoiding duplicate-content concerns. ### Source excerpt SEOmatic and a multi-environment config can prevent Google from indexing your staging sites, and diluting your SEO value ## Mention's CTO journey to 400,000 users DevFeed: [Mention's CTO journey to 400,000 users]() Original publisher: [Read original article]() Author: Hexa Published: 2015-10-22T12:17:15Z Content type: article Language: en Sources: [eFounders]() Topics: [App](), [Monitoring](), [Crawler](), [Concurrency](), [race-condition](), [Go Language](), [MySQL](), [API](), [Kafka](), [Redis](), [Percona](), [backups]() Tags: [api](), [backups](), [developer](), [development](), [dns](), [go](), [kafka](), [monitoring](), [mysql](), [percona](), [programming](), [race-condition](), [redis](), [startup](), [tech](), [users]() ### AI overview Mention's co-founder and CTO describes the technical challenges involved in growing the real-time monitoring application to 400,000 users. The article covers high-concurrency web crawling, a libc DNS resolver race condition, and the use of Go as a workaround. It also explains Mention's storage architecture using Percona MySQL, Redis for caching, Kafka for messaging, and incremental backups. ### Source excerpt Mention is a real-time monitoring application used to track and analyze trends and e-reputation in a very complete and intuitive way. Co-founded in 2010, it now counts 400,000 users across the world. This impressive growth rate not only implies great marketing talent but also impressive technical achievements. Arnaud le Blanc, Mention's co-founder and CTO, tells us about how he got Mention there. You gotta love technical challenges "When we developed Mention, our main competitor was Google Alerts, which feels a little like being David versus Goliath at first. But then it became one of the reasons I like my job: it is very challenging. Getting to this point of a product development involves a true entrepreneurial mindset. Before being Mention's CTO, I was a developer and then the lead developer working on the media monitoring feature at Pressking. eFounders offered me to become Mention's co-founder. I might not have thought of myself as an entrepreneur before, although today, I really feel like Mention is my product: I've built it together with my co-founders. Building an application like Mention is full of technical challenges. We handle a large amount of data, crawl thousands of web page per second, built an open API for our own apps, and use lots of third-party APIs." FOCUS: a challenge overcome when developing Mention's web crawler Since Mention crawls a large amount of web pages in parallel, there is a point when the crawler reaches a high concurrency level and stresses the OS: that is when we got to the limit of our system. In our case, these limits entailed the appearance of a race condition in the libc's DNS resolver. libc was sending over DNS queries on random unrelated file descriptors, which was leading to weird behaviors. The bug was reported and fixed a few months later, but in the meantime, we had to use the plain-Go DNS resolve, which happened to work really well. "At Mention, several millions of Mentions are inserted in user feeds per day (see infogra ## robots.txt usage over the Alexa million DevFeed: [robots.txt usage over the Alexa million]() Original publisher: [Read original article]() Author: ben@benjojo.co.uk Published: 2015-09-05T16:35:21Z Content type: article Language: en Sources: [benjojo blog]() Topics: [Crawler](), [Bot]() Tags: [bots](), [robots-txt]() ### AI overview The article examines how website operators use robots.txt to guide bots and crawlers, including search engines, price indexers, social-media embed bots, SEO services, and archives. It discusses the protocol's voluntary nature and its role in limiting crawling, resource use, indexing, and exposure of sensitive site paths. ### Source excerpt robots.txt usage over the Alexa million If you ever had to deal with bots while running a site you will have at least at some point looked into robots.txt, a system that isn't rea ## Jak dokuczać spamerom DevFeed: [Jak dokuczać spamerom]() Original publisher: [Read original article]() Author: Tom Published: 2012-12-18T00:00:00Z Content type: tutorial Language: pl Sources: [Tomasz Gągor]() Topics: [WordPress](), [Crawler](), [Template](), [DDoS]() Tags: [ddos](), [php](), [robots](), [spam](), [wordpress]() ### AI overview The article describes a technique for generating large numbers of fake email addresses on a WordPress page to trap spam crawlers. It recommends excluding the path in robots.txt and discusses precautions to avoid causing mail-related denial-of-service problems for one's own or third-party domains. ### Source excerpt Dawno, dawno temu... Za górami, za lasami... czytałem sobie tekst Lemat'a o dokuczaniu spamerom i pomyślałem że sam też tak mogę i nawet chcę więc popełniłem skrypcik, który dla losowych słów generował maile. Skrypcik działał z dwa lata na mojej poprzedniej stronie i nie raz zdarzyło się tam jakiejś mendzie zapętlić. Jakoś nie miałem czasu od razu, a później zapomniałem wrzucić go na nową stronie i tak zostało - na pewien czas.