# SourceHut author describes the operational cost of aggressive LLM crawlers

DevFeed: [SourceHut author describes the operational cost of aggressive LLM crawlers](<https://devfeed.tech/articles/please-stop-externalizing-your-costs-directly-into-my-face-20810.md>)

Original publisher: [Read original article](<https://drewdevault.com/blog/Stop-externalizing-your-costs-on-me/>)

Author: March

Published: 2025-03-17T00:00:00Z

Content type: opinion

Language: en

Sources: [Drew DeVault](<https://devfeed.tech/sources/drew-devault.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Crawler](<https://devfeed.tech/topics/crawler.md>), [Git](<https://devfeed.tech/topics/git.md>), [Go Language](<https://devfeed.tech/topics/go-language.md>), [ci](<https://devfeed.tech/topics/ci.md>), [Cryptocurrency](<https://devfeed.tech/topics/cryptocurrency.md>), [HTTP](<https://devfeed.tech/topics/http.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ci](<https://devfeed.tech/tags/ci.md>), [cryptocurrency](<https://devfeed.tech/tags/cryptocurrency.md>), [git](<https://devfeed.tech/tags/git.md>), [go](<https://devfeed.tech/tags/go.md>), [http](<https://devfeed.tech/tags/http.md>), [llms](<https://devfeed.tech/tags/llms.md>), [robots](<https://devfeed.tech/tags/robots.md>)

## AI overview

A personal opinion post describes SourceHut's experience mitigating aggressive LLM crawlers that ignore robots.txt, crawl expensive Git endpoints, distribute requests across many IP addresses, and contribute to recurring outages. It also compares this burden with earlier CI cryptocurrency-mining abuse and Go module mirror traffic.

## Source excerpt

This blog post is expressing personal experiences and opinions and doesn't reflect any official policies of SourceHut. Over the past few months, instead of working on our priorities at SourceHut, I have spent anywhere from 20-100% of my time in any given week mitigating hyper-aggressive LLM crawlers at scale. This isn't the first time SourceHut has been at the wrong end of some malicious bullshit or paid someone else's externalized costs - every couple of years someone invents a new way of ruining my day. Four years ago, we decided to require payment to use our CI services because it was being abused to mine cryptocurrency. We alternated between periods of designing and deploying tools to curb this abuse and periods of near-complete outage when they adapted to our mitigations and saturated all of our compute with miners seeking a profit. It was bad enough having to beg my friends and family to avoid "investing" in the scam without having the scam break into my business and trash the place every day. Two years ago, we threatened to blacklist the Go module mirror because for some reason the Go team thinks that running terabytes of git clones all day, every day for every Go project on git.sr.ht is cheaper than maintaining any state or using webhooks or coordinating the work between instances or even just designing a module system that doesn't require Google to DoS git forges whose entire annual budgets are considerably smaller than a single Google engineer's salary. Now it's LLMs. If you think these crawlers respect robots.txt then you are several assumptions of good faith removed from reality. These bots crawl everything they can find, robots.txt be damned, including expensive endpoints like git blame, every page of every git log, and every commit in every repo, and they do so using random User-Agents that overlap with end-users and come from tens of thousands of IP addresses - mostly residential, in unrelated subnets, each one making no more than one HTTP request ove