# SWHForest: A Structural Curation and Search Layer for Large-Scale Code Corpora

DevFeed: [SWHForest: A Structural Curation and Search Layer for Large-Scale Code Corpora](<https://devfeed.tech/articles/swhforest-a-structural-curation-and-search-layer-for-large-scale-code-corpora-125379.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11439>)

Author: Federico Pennino, Andrea Gurioli, Maurizio Gabbrielli, Paolo Ferragina

Published: 2026-11-10T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [code search](<https://devfeed.tech/topics/code-search.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [archive](<https://devfeed.tech/tags/archive.md>), [ast](<https://devfeed.tech/tags/ast.md>), [auditing](<https://devfeed.tech/tags/auditing.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [boilerplate](<https://devfeed.tech/tags/boilerplate.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [curation](<https://devfeed.tech/tags/curation.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [download](<https://devfeed.tech/tags/download.md>), [downloads](<https://devfeed.tech/tags/downloads.md>), [file](<https://devfeed.tech/tags/file.md>), [files](<https://devfeed.tech/tags/files.md>), [framework](<https://devfeed.tech/tags/framework.md>), [index](<https://devfeed.tech/tags/index.md>), [internet](<https://devfeed.tech/tags/internet.md>), [intersection](<https://devfeed.tech/tags/intersection.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [layer](<https://devfeed.tech/tags/layer.md>), [search](<https://devfeed.tech/tags/search.md>)

## AI overview

SWHForest adds a structural search layer to pointer-based code corpora so researchers can identify files to download without first retrieving and parsing the full archive. It parses each file into an AST in one pass, discards the source text, and retains a structural summary. Queries run locally with SQL or index intersection, or can be exposed by archive providers as a server-side filtering API; only matching candidates are then downloaded. On the ten largest Stack-Edu languages (52.1 million unique files and 341 million definitions), the authors report millisecond query times and 46-72% fewer downloads. They also report that live structural searches fell from hours of per-language streaming to seconds or minutes.

## Source excerpt

arXiv:2611.11439v1 Announce Type: cross Abstract: Pre-training code language models relies on massive code corpora usually streamed over the internet. However, researchers rarely need a multi-terabyte archive in its entirety; instead, their workflow centers on curated subsets: selecting well-documented files, filtering out repetitive boilerplate, auditing for benchmark contamination, or retrieving specific code patterns. Yet, because modern corpora are typically published as file pointers and basic attributes rather than source text, no general structural layer exists to select which files to download: curating a subset requires downloading and parsing millions of files first just to discard most of them. We present SWHForest, an open-source framework and public dataset that provides this missing structural layer for any pointer-based corpus without redistributing source code. A single pass over the archive parses each file into an AST and immediately discards the source text, keeping only its structural summary. Queried locally or exposed by archive providers as a server-side filtering API, SWHForest selects matching files first (via SQL or index intersection) and downloads only surviving candidates. Across the ten largest languages of Stack-Edu (52.1M unique files, 341M definitions), SWHForest answers curation queries in milliseconds while cutting downloads by 46-72%, reducing live structural searches from hours of per-language streaming to seconds or minutes.