# Scaling AI-based Data Processing with Hugging Face + Dask

DevFeed: [Scaling AI-based Data Processing with Hugging Face + Dask](<https://devfeed.tech/articles/scaling-ai-based-data-processing-with-hugging-face-dask-7158.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/dask-scaling>)

Author: Sarah Johnson; James Bourbeau; Quentin Lhoest; Daniel van Strien

Published: 2024-10-09T00:00:00Z

Content type: tutorial

Language: en

Sources: [Hugging Face - Blog](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [data-processing](<https://devfeed.tech/topics/data-processing.md>), [datasets](<https://devfeed.tech/topics/datasets.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [pandas](<https://devfeed.tech/topics/pandas.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [anthropic](<https://devfeed.tech/topics/anthropic.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Python](<https://devfeed.tech/topics/python.md>)

Tags: [data-processing](<https://devfeed.tech/tags/data-processing.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [guide](<https://devfeed.tech/tags/guide.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [inference](<https://devfeed.tech/tags/inference.md>), [information-retrieval](<https://devfeed.tech/tags/information-retrieval.md>), [nlp](<https://devfeed.tech/tags/nlp.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [python](<https://devfeed.tech/tags/python.md>), [scale](<https://devfeed.tech/tags/scale.md>)

## AI overview

This tutorial demonstrates scaling AI-based data processing with Hugging Face and Dask. It processes the FineWeb dataset locally with pandas, then scales text classification to 211 million rows across multiple cloud GPUs using Dask DataFrame and Parquet files.

## Source excerpt

We're on a journey to advance and democratize artificial intelligence through open source and open science.