# Web Scraping A method of extracting data from websites, typically through automated software such as bots or web crawlers. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Claims About RubyGems Caching and RubyDoc Code Execution DevFeed: [Claims About RubyGems Caching and RubyDoc Code Execution]() Original publisher: [Read original article]() Published: 2026-09-12T00:02:11Z Content type: opinion Language: en Sources: [Aaron Patterson]() Topics: [rubygems](), [Caching](), [vulnerability](), [Docker](), [Web Scraping]() Tags: [caching](), [docker](), [rubygems](), [scraping](), [vulnerability]() ### AI overview An opinion post discusses alleged RubyGems activity involving a caching vulnerability and web scraping. It also describes how published gems can cause RubyDoc.info to run YARD-loaded code in a networked Docker container. ### Source excerpt Today Reuters and the Wall Street Journal both reported about rogue AI agents at OpenAI attacking RubyGems.org. https://www.rubyhack.ai/ has an amazing writeup, and you should read it. I just wanted to make a quick post about it because it's wild. TL;DR: It seems like OpenAI Bots knew about the RubyGems caching vulnerability, tried to take advantage of it, and at the same time ran some weird web scraping code on RubyDoc.info. Back in May, socket.dev reported about a "GemStuffer Campaign" where someone (I guess OpenAI) was uploading tons of junk gems to RubyGems.org. For some reason, the gems would scrape UK government websites, then repackage the data as gems, and attempt to upload them to RubyGems. I honestly didn't think much about this (or even look into it) until Sydney Von Arx and Spencer Kitts (both co-authors on https://www.rubyhack.ai) contacted me asking about RubyGems. I thought the claims they were making were completely outlandish until I actually read the code in these "GemStuffer" gems. After reading the code in these gems, a couple things stood out to me. YARD Documentation First, the gems leverage YARD documentation to execute arbitrary code on host machines. In most of the examples you'll see a .yardopts file that looks like this: --load ./script.rb README.md lib/**/*.rb Here's a link to an example. If you have YARD installed, and you install this gem, then YARD will load and run whatever is in ./script.rb from inside the gem. I think it's pretty common knowledge that C extensions will execute extconf.rb (so you basically have an RCE vector), but I was surprised to find out that a documentation tool would do that too. Nobody is going to install a gem named slnleaker5 though, so why would this matter? Well, any time a Gem is published RubyDoc.info will download the gem and process the YARD documentation. RubyDoc.info will execute the arbitrary code inside a Docker container. The Docker container still has network access though, so these gems could ha ## The Real Python Podcast - Episode #304: Configuring a Versatile LLM Harness & Scraping the Web With Scrapy DevFeed: [The Real Python Podcast - Episode #304: Configuring a Versatile LLM Harness & Scraping the Web With Scrapy]() Original publisher: [Read original article]() Author: Real Python Published: 2026-07-24T12:00:00Z Content type: article Language: en Sources: [Real Python]() Topics: [Large Language Model](), [Python](), [Web Scraping](), [Homelab](), [Artificial Intelligence](), [data](), [API]() Tags: [agentic](), [ai](), [api](), [applications](), [developer](), [hosting](), [llm](), [podcast](), [python](), [self-hosting](), [web](), [workflow]() ### AI overview Real Python Podcast Episode 304 discusses building versatile harnesses around large language models, efficient agentic developer workflows, web scraping with Scrapy, and self-hosting Python applications. The episode also covers local models, edge computing, token use, tools, and related projects. ### Source excerpt Which is more important, the model or the "harness" around an LLM? What are ways to assemble an efficient agentic developer workflow? This week on the show, Ayan Pahwa joins us to discuss harnessing, web scraping, and self-hosting Python applications. ## Vercel Functions can now run up to 30 minutes DevFeed: [Vercel Functions can now run up to 30 minutes]() Original publisher: [Read original article]() Author: Tiago Ventura Loureiro Published: 2026-06-15T14:00:00Z Content type: release Language: en Sources: [Vercel News]() Topics: [Vercel](), [Node.js](), [Python](), [Artificial Intelligence](), [Large Language Model](), [Next.js](), [Automation](), [Headless Browser](), [Web Scraping]() Tags: [ai](), [apis](), [automation](), [browser](), [cost](), [llm](), [node-js](), [python](), [vercel](), [web]() ### AI overview Vercel Functions running on Node.js and Python can now execute for up to 30 minutes for Pro and Enterprise teams. The update supports longer-running workloads such as LLM reasoning, streamed AI responses, document and media processing, OCR, web scraping, browser automation, workflow steps, and queue handlers. ### Source excerpt Vercel Functions using the Node.js and Python runtimes now support execution durations up to 30 minutes for Pro and Enterprise teams, more than 2x the previous 800 second limit. Support for additional runtimes is coming soon. Use longer-running Functions for work that needs more time to finish, including: Long LLM reasoning and tool calls AI responses that stream for several minutes Document and media processing OCR and extraction Web scraping and browser automation Complex Workflow steps or Queue handlers Fluid compute keeps long-running work cost-efficient. Active CPU billing only applies while your code is executing, and pauses while your Function is waiting on I/O such as AI model calls, database queries, and third-party APIs. Set maxDuration to opt in. For Next.js App Router, configure it in the route file: For other runtimes and frameworks, configure maxDuration for a specific function path in vercel.json: Durations above 800 seconds are in beta and require Fluid compute. Learn more about configuring max duration for Vercel Functions in the documentation. Read more ## One honojs file for entire web scraping API DevFeed: [One honojs file for entire web scraping API]() Original publisher: [Read original article]() Author: iHateReading Published: 2026-04-13T16:54:35Z Content type: tutorial Language: en Sources: [iHateReading]() Topics: [Hono](), [Web Scraping](), [API]() Tags: [api](), [backend](), [blockchain](), [coding](), [development](), [frontend](), [github-repository](), [honojs](), [ihatereading](), [javascript](), [node-js](), [one-honojs-file-for-entire-web-scraping-api](), [product](), [programming](), [react](), [scraping](), [typescript](), [web-development](), [web-scraping](), [web-scraping-github-repository-scraping-backend-honojs]() ### AI overview A tutorial about building web-scraping endpoints in a single Honojs file. ### Source excerpt Building scrape endpoints using one honojs file ## Data Is the Bottleneck in Many AI Projects DevFeed: [Data Is the Bottleneck in Many AI Projects]() Original publisher: [Read original article]() Author: Brian Jenney Published: 2026-02-07T16:25:26Z Content type: opinion Language: en Sources: [Brian Jenney]() Topics: [data](), [Web Scraping](), [Structured-data](), [Artificial Intelligence]() Tags: [ai](), [data](), [data-pipeline](), [grounding](), [web-scraping]() ### AI overview The article argues that data collection and preparation, rather than wiring together AI components, are often the main bottlenecks in AI projects. It discusses messy real-world data, data pipelines, vector databases, and web scraping as ways to ground useful AI outputs. ### Source excerpt This is the first Saturday I've had off in a few weeks. ## How Grepsr uses Temporal to deliver scalable and reliable web data DevFeed: [How Grepsr uses Temporal to deliver scalable and reliable web data]() Original publisher: [Read original article]() Author: Subrat Basnet Published: 2025-07-24T00:00:00Z Content type: article Language: en Sources: [Temporal Blog]() Topics: [data](), [Web Scraping](), [reliability](), [Microservices](), [Orchestration](), [event driven](), [Monitoring](), [Architecture & Design](), [retry]() Tags: [architecture](), [code](), [community](), [data](), [event-driven](), [microservices](), [monitoring](), [orchestration](), [reliability](), [retry](), [web]() ### AI overview Grepsr describes how it uses Temporal to orchestrate a microservices-based, event-driven web-data extraction platform. The article explains how workflow orchestration addresses failure tracing, lost state, brittle parallel execution, and limited cron-job visibility while supporting large-scale data processing. ### Source excerpt Grepsr scaled to 600 M web-data records a day with 99% delivery reliability by orchestrating its micro-services on Temporal -- see how code-first Workflows made it happen. ## Building RagRabbit, An Open Source RAG Search with Postgres as the Vector Store DevFeed: [Building RagRabbit, An Open Source RAG Search with Postgres as the Vector Store]() Original publisher: [Read original article]() Author: Carlota Soto Published: 2025-03-17T17:40:19Z Content type: article Language: en Sources: [Blog -- Neon Docs]() Topics: [Retrieval Augmented Generation (RAG)](), [Databases](), [Web Scraping](), [llamaindex](), [Model Context Protocol (MCP)](), [Appwrite](), [Claude](), [Vercel](), [Serverless]() Tags: [ai](), [building](), [case-studies](), [claude](), [cursor](), [databases](), [embedding](), [llamaindex](), [mcp](), [openai](), [rag](), [search](), [serverless](), [source](), [vercel]() ### AI overview This article introduces RagRabbit, an open-source toolkit for building retrieval-augmented generation workflows with Postgres and pgVector. It crawls websites, converts pages to Markdown, creates LLM-friendly text files, stores embeddings in Postgres, and provides AI question answering through OpenAI or Claude. It also offers an MCP server for supplying relevant document chunks to Cursor and Claude Desktop, with deployment on Vercel and Neon. ### Source excerpt "When I started RagRabbit, I did testing on vector databases, but I didn't see a real advantage. Postgres with Pgvector covers everything I need, and it's very performant for the number of rows I handle" (Marco D'Alia, Software Architect behind RagRabbit) While experimenting with... ## Fighting TLS fingerprinting with Node.js DevFeed: [Fighting TLS fingerprinting with Node.js]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2021-12-07T13:40:00Z Content type: tutorial Language: en Sources: [HTTP Toolkit]() Topics: [TLS (Transport Layer Security)](), [Node.js](), [Web Scraping](), [HTTP]() Tags: [browser](), [http](), [interception](), [javascript](), [mitm](), [node-js](), [proxy](), [tls]() ### AI overview This tutorial explains how TLS fingerprinting lets servers and proxies identify client software from unencrypted connection metadata. It examines how this can block HTTP Toolkit and other automated traffic, then shows how Node.js techniques can work around that blocking. ### Source excerpt The modern internet is full of services that want to know who you are. Fingerprinting is the latest way to do this: capturing many small details about your client, and using it to create an id that's sufficiently unique to recognize you and infer details about your network client and device. This is a privacy problem, which I'm not going to focus on here, but collecting and analysing interaction metadata is also a powerful tool to recognize certain types of clients, even when they attempt to disguise themselves. TLS provides a particularly good surface for this kind of fingerprinting, which allows a server or proxy to recognize the kind of software (a specific browser version, Python, Ruby, Node.js, etc) that's opening any TLS connection, before the client has even sent any data (such as an HTTP request) within the connection, and purely using unencrypted public data from the connection. In many cases, this is a problem. HTTP Toolkit acts as a MitM proxy for HTTP(S) traffic inspection & mocking, and this potentially allows servers to recognize and block it, along with any other similar debugging proxies. Many other automated scripts and tools can also be recognized, blocking web scraping and other requests from anything but a real browser. Until recently, I thought this was fairly theoretical, but last week an HTTP Toolkit user showed me a real-world example, where non-browser traffic is blocked completely, based just on its TLS fingerprint, causing big problems for HTTP Toolkit usage. Fortunately, we can work around this. In this article, I want to explain how TLS fingerprinting works, look at a real-world example, and then see exactly how you can defeat this blocking using Node.js (with techniques that you can easily apply elsewhere too). How does TLS fingerprinting work? TLS provides a huge amount of data for fingerprinting. Every connection secured by TLS (for example, all HTTPS requests) starts with a 'client hello' message from the client, sent unencrypted, wi