# We Stopped Feeding LLMs Raw HTML: Cutting Agent Scraping Costs by 74% Across 10,000 Targets

DevFeed: [We Stopped Feeding LLMs Raw HTML: Cutting Agent Scraping Costs by 74% Across 10,000 Targets](<https://devfeed.tech/articles/we-stopped-feeding-llms-raw-html-cutting-agent-scraping-costs-by-74-across-10-000-targets-117857.md>)

Original publisher: [Read original article](<https://hackernoon.com/we-stopped-feeding-llms-raw-html-cutting-agent-scraping-costs-by-74percent-across-10000-targets?source=rss>)

Author: Hamed Hosseini

Published: 2026-11-04T21:41:41Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>), [Bot](<https://devfeed.tech/topics/bot.md>), [performance-optimization](<https://devfeed.tech/topics/performance-optimization.md>), [Agent Browser](<https://devfeed.tech/topics/agent-browser.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [HTML](<https://devfeed.tech/topics/html.md>), [Markdown](<https://devfeed.tech/topics/markdown.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [403](<https://devfeed.tech/tags/403.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-scraping-agents](<https://devfeed.tech/tags/ai-scraping-agents.md>), [all](<https://devfeed.tech/tags/all.md>), [api](<https://devfeed.tech/tags/api.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [automation](<https://devfeed.tech/tags/automation.md>), [bot](<https://devfeed.tech/tags/bot.md>), [costs](<https://devfeed.tech/tags/costs.md>), [data](<https://devfeed.tech/tags/data.md>), [fail](<https://devfeed.tech/tags/fail.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [html](<https://devfeed.tech/tags/html.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [llms](<https://devfeed.tech/tags/llms.md>), [production](<https://devfeed.tech/tags/production.md>), [proxy](<https://devfeed.tech/tags/proxy.md>), [raw](<https://devfeed.tech/tags/raw.md>), [routing](<https://devfeed.tech/tags/routing.md>), [scraping](<https://devfeed.tech/tags/scraping.md>), [token](<https://devfeed.tech/tags/token.md>), [web-scraping](<https://devfeed.tech/tags/web-scraping.md>)

## AI overview

The article reports that a three-tier scraper raised success across 10,000 bot-protected URLs from 36.4% to 99.4% while cutting cost per 10,000 operations from $184.20 to $47.50. It tries free direct HTTP/2 first, uses Bright Data Web Unlocker for blocked or challenged requests, and reserves Bright Data Scraping Browser for pages needing client-side rendering. A preprocessing pipeline harvests JSON-LD product data without an LLM call when available; otherwise it removes low-value markup and formats dense content as Markdown tables. The reported changes also reduced worker memory from 640 MB to 38 MB, median latency from 14.8 to 2.6 seconds, and tokens per call from 14,200 to 1,650. Remaining issues included session lockouts, WebGL challenges, and cascading retries; the authors addressed these with session pinning, selective cloud-browser use, and backoff with cached snapshots.

## Source excerpt

Most AI scraping agents fail in production from anti-bot walls, token-burning HTML dumps, and proxy costs. A tri-tier routing architecture that fixed all three.