# Data pipelines with Apache Crunch and Java 8

DevFeed: [Data pipelines with Apache Crunch and Java 8](<https://devfeed.tech/articles/data-pipelines-with-apache-crunch-and-java-8-2021.md>)

Original publisher: [Read original article](<https://developers.soundcloud.com/blog//data-pipelines-apache-crunch-java-8>)

Published: 2016-06-01T00:00:00Z

Content type: article

Language: en

Sources: [SoundCloud Backstage Blog](<https://devfeed.tech/sources/soundcloud-backstage-blog.md>)

Topics: [data-processing](<https://devfeed.tech/topics/data-processing.md>)

Tags: [apache](<https://devfeed.tech/tags/apache.md>), [api](<https://devfeed.tech/tags/api.md>), [batch](<https://devfeed.tech/tags/batch.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [clojure](<https://devfeed.tech/tags/clojure.md>), [crunch](<https://devfeed.tech/tags/crunch.md>), [data](<https://devfeed.tech/tags/data.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [developer](<https://devfeed.tech/tags/developer.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [errors](<https://devfeed.tech/tags/errors.md>), [event](<https://devfeed.tech/tags/event.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [insights](<https://devfeed.tech/tags/insights.md>), [java](<https://devfeed.tech/tags/java.md>), [jobs](<https://devfeed.tech/tags/jobs.md>), [learning](<https://devfeed.tech/tags/learning.md>), [mapping](<https://devfeed.tech/tags/mapping.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [scala](<https://devfeed.tech/tags/scala.md>)

## AI overview

The article explains how SoundCloud uses Apache Crunch and its Crunch Lambda module with Java 8 for early-stage, high-volume batch data-pipeline processing. It argues that Hadoop MapReduce remains suitable for robust dataset transformation, cleaning, and partitioning, and highlights Crunch's type-safe API for structured Protobuf records.

## Source excerpt

With Java 8 now in the mainstream, Scala and Clojure are no longer the only choices to develop readable, functional code for big data technology on the JVM. In this post we see how SoundCloud is leveraging Apache Crunch and the new Crunch Lambda module to do the high-volume data processing tasks which are essential at early stages in our batch data pipeline efficiently, robustly and simply in Java 8.