# llms.txt

Published articles for llms.txt.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Cloudflare update distinguishes search crawlers from AI training bots

DevFeed: [Cloudflare update distinguishes search crawlers from AI training bots](<https://devfeed.tech/articles/cloudflare-40899.md>)

Original publisher: [Read original article](<https://habr.com/ru/news/1083142/>)

Author: neuromamontov

Published: 2026-09-16T21:09:58Z

Content type: news

Language: ru

Sources: [Tagir Valeev](<https://devfeed.tech/sources/tagir-valeev.md>)

Topics: [Cloudflare](<https://devfeed.tech/topics/cloudflare.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [2025](<https://devfeed.tech/tags/2025.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-5ebc2dd0d80c](<https://devfeed.tech/tags/ai-5ebc2dd0d80c.md>), [cloudflare](<https://devfeed.tech/tags/cloudflare.md>), [geo](<https://devfeed.tech/tags/geo.md>), [llms-txt](<https://devfeed.tech/tags/llms-txt.md>), [robots-txt](<https://devfeed.tech/tags/robots-txt.md>), [tag-3ec72a25b0de](<https://devfeed.tech/tags/tag-3ec72a25b0de.md>), [tag-6dbce23d3386](<https://devfeed.tech/tags/tag-6dbce23d3386.md>), [tag-ba47f5ee56ab](<https://devfeed.tech/tags/tag-ba47f5ee56ab.md>), [tag-c94f063f9c70](<https://devfeed.tech/tags/tag-c94f063f9c70.md>), [tag-d5d895f883db](<https://devfeed.tech/tags/tag-d5d895f883db.md>)

### AI overview

The article discusses Cloudflare's update for distinguishing search crawlers from AI training bots. It explains why website owners may want to allow search indexing while restricting AI training access, and notes that the earlier llms.txt approach was not widely adopted.

### Source excerpt

Буду первым, кто не просто перепостит новость про обновление Cloudflare, а ещё и по существу прокомментирует, почему тема разделения ботов на поисковых и обучающих так важна для владельцев сайтов. Что показательно, тема ещё и совсем не новая: эти разговоры я слышал ещё года полтора назад -- тогда все обсуждали формат llms.txt, который, к слову, так толком и не взлетел. Но ключевая идея была очевидна уже тогда: желающих закрыть свой сайт от обучающих ботов в разы больше чем желающих закрыть сайт от поисковых ботов -- 17% против менее чем 1% (оценка самого Cloudflare, 2025). Читать далее

## Highlights from the 2026 State of Knowledge Report

DevFeed: [Highlights from the 2026 State of Knowledge Report](<https://devfeed.tech/articles/highlights-from-the-2026-state-of-knowledge-report-41365.md>)

Original publisher: [Read original article](<https://www.mintlify.com/blog/the-state-of-knowledge-2026-highlights>)

Author: Lauren Volpi

Published: 2026-09-16T00:00:00Z

Content type: article

Language: en

Sources: [Mintlify Blog](<https://devfeed.tech/sources/mintlify-blog.md>)

Topics: [Documentation](<https://devfeed.tech/topics/documentation.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [HTML](<https://devfeed.tech/topics/html.md>), [Website](<https://devfeed.tech/topics/website.md>), [MCP](<https://devfeed.tech/topics/mcp.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-trends](<https://devfeed.tech/tags/ai-trends.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [llms-txt](<https://devfeed.tech/tags/llms-txt.md>), [markdown](<https://devfeed.tech/tags/markdown.md>), [mcp](<https://devfeed.tech/tags/mcp.md>), [report](<https://devfeed.tech/tags/report.md>)

### AI overview

Mintlify's 2026 State of Knowledge Report examines how AI agents are changing documentation readership and access. It reports that agent requests exceeded human page loads on Mintlify-powered sites in August, with many agents using Markdown, llms.txt, agent skills, and MCP tool calls. Controlled tests across 20 documentation sites found that linking Markdown pages to llms.txt improved agent navigation and reduced token consumption.

### Source excerpt

AI agents overtook humans as the primary readers across Mintlify-powered sites. In August alone, agents made 257 million web requests compared to 131 million human page loads, which means agent traffic now outnumbers human traffic by nearly 2:1.