# Too Big to Inspect? Ethically Auditing Large Datasets Through Small Samples

DevFeed: [Too Big to Inspect? Ethically Auditing Large Datasets Through Small Samples](<https://devfeed.tech/articles/too-big-to-inspect-ethically-auditing-large-datasets-through-small-samples-125665.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10399>)

Author: Giovana Marchetti, Jhessica Silva, Tiago Torrent, Carlos Caetano, Sandra Avila

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [big-data](<https://devfeed.tech/topics/big-data.md>), [human feedback](<https://devfeed.tech/topics/human-feedback.md>), [ai regulation](<https://devfeed.tech/topics/ai-regulation.md>)

Tags: [accountability](<https://devfeed.tech/tags/accountability.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-ethics](<https://devfeed.tech/tags/ai-ethics.md>), [ai-regulation](<https://devfeed.tech/tags/ai-regulation.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [auditing](<https://devfeed.tech/tags/auditing.md>), [automated](<https://devfeed.tech/tags/automated.md>), [bias](<https://devfeed.tech/tags/bias.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [data](<https://devfeed.tech/tags/data.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [datasheets](<https://devfeed.tech/tags/datasheets.md>), [ethics](<https://devfeed.tech/tags/ethics.md>), [human-resources](<https://devfeed.tech/tags/human-resources.md>), [industry](<https://devfeed.tech/tags/industry.md>), [large-datasets](<https://devfeed.tech/tags/large-datasets.md>), [models](<https://devfeed.tech/tags/models.md>), [problems](<https://devfeed.tech/tags/problems.md>), [process](<https://devfeed.tech/tags/process.md>), [small](<https://devfeed.tech/tags/small.md>)

## AI overview

The paper finds that random samples can support qualitative ethical audits of large datasets without examining every item. It applied four AI Ethics Tools--Datasheets for Datasets, Data Statements, Data Nutrition Labels, and Data Cards--to a 1% sample of Framed Multi30K, and Datasheets for Datasets to two 0.00006% samples of relaion2B-en-research-safe. The authors report that these audits surfaced ethical problems, structural biases, and technical inconsistencies with manageable human and computational resources. The tools' questions prompted different perspectives, but analysis and findings depended fundamentally on human judgment. The abstract describes these case studies as evidence that small-sample audits are feasible; it does not establish that sampling captures every issue in a dataset.

## Source excerpt

arXiv:2611.10399v1 Announce Type: cross Abstract: Big techs and researchers increasingly claim that their training datasets are too large to inspect for bias and ethical problems. However, in the race to build ever-large models, they have relied on automated web scraping and no human curation. Datasets built this way absorb the stereotypes, exclusionary norms, and historical prejudices of their sources, and the models trained on them can reproduce and amplify those patterns. Because manually auditing billions of samples is humanly impossible, we need practical, effective methods to identify biases in these datasets. This paper studies the feasibility of ethically auditing large datasets through small samples and proposes a practical method for identifying structural biases by applying AI Ethics Tools (AIETs) to random samples. We applied Datasheets for Datasets, Data Statements, Data Nutrition Labels, and Data Cards to a 1% sample of Framed Multi30K, and Datasheets for Datasets to two 0.00006% samples of relaion2B-en-research-safe. Our results show that evaluating small samples with AIETs can reveal ethical problems, structural biases, and technical inconsistencies. Although industry practice assumes that identifying problems requires analyzing the entire dataset, we show that a random subset supports a viable qualitative analysis without overwhelming computational or human resources. Demonstrating that a dataset is biased does not require examining every instance: if a bias appears in a random sample, it is a flaw of the creation process. The AIETs' questions acted as provocateurs, evoking different perspectives and analysis methods for the same dataset, but the analysis and its results depended fundamentally on the human factor. We hope these findings contribute to urgent public discussions about ethics, transparency, accountability, and AI regulation by offering a practical pathway for responsibly auditing massive datasets.