# Scalding

Scalding is a Scala library for specifying Hadoop MapReduce jobs, built on the Cascading Java library.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Running Apache Flink on Amazon Elastic Mapreduce

DevFeed: [Running Apache Flink on Amazon Elastic Mapreduce](<https://devfeed.tech/articles/running-apache-flink-on-amazon-elastic-mapreduce-20847.md>)

Original publisher: [Read original article](<http://themodernlife.net/scala/hadoop/hdfs/sclading/flink/streaming/realtime/emr/aws/2016/01/06/running-apache-flink-on-amazon-elastic-mapreduce/>)

Published: 2016-01-06T08:15:13Z

Content type: tutorial

Language: en

Sources: [Ian Hummel](<https://devfeed.tech/sources/ian-hummel.md>)

Topics: [apache-flink](<https://devfeed.tech/topics/apache-flink.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [Scalding](<https://devfeed.tech/topics/scalding.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>)

Tags: [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [aws](<https://devfeed.tech/tags/aws.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [ec2](<https://devfeed.tech/tags/ec2.md>), [emr](<https://devfeed.tech/tags/emr.md>), [flink](<https://devfeed.tech/tags/flink.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [hdfs](<https://devfeed.tech/tags/hdfs.md>), [linux](<https://devfeed.tech/tags/linux.md>), [realtime](<https://devfeed.tech/tags/realtime.md>), [s3](<https://devfeed.tech/tags/s3.md>), [scala](<https://devfeed.tech/tags/scala.md>), [scalding](<https://devfeed.tech/tags/scalding.md>), [sclading](<https://devfeed.tech/tags/sclading.md>), [spark](<https://devfeed.tech/tags/spark.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

A tutorial on running Apache Flink jobs on Amazon EMR, including adapting Scalding jobs and launching and accessing an EMR cluster. It also discusses EMRFS support for reading data from Amazon S3.

### Source excerpt

I love really Amazon EMR. Over the years it's grown from being "Hadoop on-demand" to a full-fledged cluster management system for running OSS big-data apps (Hadoop MR of course, but also Spark, Hue, Hive, Pig, Oozie and more). While Hadoop out of the box supports reading from S3, EMR has a proprietary implementation called EMRFS that has some nice features. For those reasons, it's really the best Hadoop cluster to use if you're storing your data in S3. Lately I've been experimenting a lot with Apache Flink to replace MR as the excution fabric. At work, we have many, many jobs written in Scalding. Flink can execute Scalding jobs with some very simple modifications which was a great way to move our jobs from MR to a more memory-centric data processing model. However, we really wanted to run our jobs on EMR using Flink. Flink is not an option for EMR (yet) but can we still get our jobs to run? Let's see! Start an EMR cluster The first thing we need is an EMR cluster. You can launch a small test cluster for very cheap. Once it's running, let's ssh onto it and see what's going on. ihummel at mm-mac-3270 in ~ $ ssh hadoop@ec2-54-226-25-85.compute-1.amazonaws.com Last login: Wed Jan 6 16:33:01 2016 from 172.85.47.138 __| __|_ ) _| ( / Amazon Linux AMI ___|\___|___| https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/ 23 package(s) needed for security, out of 49 available Run "sudo yum update" to apply all updates. EEEEEEEEEEEEEEEEEEEE MMMMMMMM MMMMMMMM RRRRRRRRRRRRRRR E::::::::::::::::::E M:::::::M M:::::::M R::::::::::::::R EE:::::EEEEEEEEE:::E M::::::::M M::::::::M R:::::RRRRRR:::::R E::::E EEEEE M:::::::::M M:::::::::M RR::::R R::::R E::::E M::::::M:::M M:::M::::::M R:::R R::::R E:::::EEEEEEEEEE M:::::M M:::M M:::M M:::::M R:::RRRRRR:::::R E::::::::::::::E M:::::M M:::M:::M M:::::M R:::::::::::RR E:::::EEEEEEEEEE M:::::M M:::::M M:::::M R:::RRRRRR::::R E::::E M:::::M M:::M M:::::M R:::R R::::R E::::E EEEEE M:::::M MMM M:::::M R:::R R::::R EE:::::EEEEEEEE::::E

## Getting started with Scalding

DevFeed: [Getting started with Scalding](<https://devfeed.tech/articles/getting-started-with-scalding-20845.md>)

Original publisher: [Read original article](<http://themodernlife.net/scala/hadoop/hdfs/sclading/2015/12/20/getting-started-with-scalding/>)

Published: 2015-12-20T08:15:13Z

Content type: tutorial

Language: en

Sources: [Ian Hummel](<https://devfeed.tech/sources/ian-hummel.md>)

Topics: [Scalding](<https://devfeed.tech/topics/scalding.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [Scala](<https://devfeed.tech/topics/scala.md>), [Testing](<https://devfeed.tech/topics/testing.md>), [Compression](<https://devfeed.tech/topics/compression.md>), [Logging](<https://devfeed.tech/topics/logging.md>)

Tags: [beginners](<https://devfeed.tech/tags/beginners.md>), [build](<https://devfeed.tech/tags/build.md>), [building](<https://devfeed.tech/tags/building.md>), [compression](<https://devfeed.tech/tags/compression.md>), [getting-started](<https://devfeed.tech/tags/getting-started.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [hdfs](<https://devfeed.tech/tags/hdfs.md>), [logging](<https://devfeed.tech/tags/logging.md>), [scala](<https://devfeed.tech/tags/scala.md>), [scalding](<https://devfeed.tech/tags/scalding.md>), [sclading](<https://devfeed.tech/tags/sclading.md>), [testing](<https://devfeed.tech/tags/testing.md>), [unit-testing](<https://devfeed.tech/tags/unit-testing.md>)

### AI overview

A practical guide to starting Scalding projects with SBT. It presents a minimal project structure and dependencies, explains local Hadoop execution and logging, and shows how to test jobs locally before submitting them to a cluster.

### Source excerpt

I've been using Scalding for the last few years and really love how simple it makes writing scalalbe data processing jobs. I think many of the issues beginners have with Scalding relate to project setup. I hope this post simplifies things for people so they can started with less hassle. Building your project with SBT The official getting started guide is pretty terse. For our purposes, all you need to get started is a very simple SBT project. simple-scalding-example/ |-- build.sbt |-- project | |-- build.properties | `-- plugins.sbt |-- src | `-- main | `-- scala | `-- WordCountJob.scala `-- log4j.properties The most important thing the setup of the build.sbt file. Our example has minimal dependencies: libraryDependencies ++= Seq( "com.twitter" %% "scalding-core" % "0.15.0", "org.apache.hadoop" % "hadoop-client" % "2.2.0" % "provided", "org.slf4j" % "slf4j-log4j12" % "1.7.13" % "provided" ) The hadoop-client jar is included for compilation, unit testing and running locally on your laptop but will not be pacakged into the final "fat jar" assembly shipped out to the cluster. Additionally, a logging framework is included so that we get good logs when debugging locally. A simple log4j.properties file is included in the repo. The whole job can be run locally using real Hadoop libraries without any additional software install: $ sbt 'run net.themodernlife.WordCount --hdfs --input build.sbt --output target/output' It's even possible to get things like LZO-compression to work with this setup. The main win here is that there is a very simple, repeatable development environment that can be used to test all aspects of the job locally before sending things off to the cluster.

## Running Scalding jobs on Apache Flink

DevFeed: [Running Scalding jobs on Apache Flink](<https://devfeed.tech/articles/running-scalding-jobs-on-apache-flink-20846.md>)

Original publisher: [Read original article](<http://themodernlife.net/scala/hadoop/hdfs/sclading/flink/streaming/realtime/2015/12/20/running-scalding-jobs-on-apache-flink/>)

Published: 2015-12-20T08:15:13Z

Content type: tutorial

Language: en

Sources: [Ian Hummel](<https://devfeed.tech/sources/ian-hummel.md>)

Topics: [Scalding](<https://devfeed.tech/topics/scalding.md>), [apache-flink](<https://devfeed.tech/topics/apache-flink.md>), [Scala](<https://devfeed.tech/topics/scala.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>)

Tags: [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [flink](<https://devfeed.tech/tags/flink.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [hdfs](<https://devfeed.tech/tags/hdfs.md>), [realtime](<https://devfeed.tech/tags/realtime.md>), [scala](<https://devfeed.tech/tags/scala.md>), [scalding](<https://devfeed.tech/tags/scalding.md>), [sclading](<https://devfeed.tech/tags/sclading.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

This tutorial explains how to run Scalding jobs on Apache Flink. It outlines the required compatibility work, including Cascading 3 support, a Scalding version compiled against Cascading 3, selection of hadoop2-flink as the Cascading fabric, and a fix for a Twitter Chill bug. It also provides a sample Scala word-count job and build configuration changes.

### Source excerpt

My previous post showed a very simple Scalding workflow. Apache Flink is a real time streaming framework that's very promising. It also supports running Cascading workflows with very little modification. Surely there must be some way to run a Scalding job on top of Flink? Turns out... YES! In a nutshell Here are the high-level things we need to solve for We need a Scalding job to test this out with cascading-flink requires Cascading 3 We need a new version of Scalding - Compiled against Cascading 3 - Allows hadoop2-flink to be selected as the Cascading "fabric" There is a bug in Twitter Chill We need Flink packaged with the Chill fix Scalding job Let's start with a very simple Scalding job. You can download https://github.com/themodernlife/simple-scalding-example for some inspiration. package net.themodernlife import com.twitter.scalding._ class WordCount(args: Args) extends Job(args) { def tokenize(text: String): Array[String] = { text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("\\s+") } val input = args("input") val output = args("output") TextLine(args("input")) .flatMap[String, String]('line -> 'word)(tokenize) .groupBy('word)(_.size) .write(Tsv(output)) } We'll be making some updates to our build.sbt. Here's what we're starting with organization := "net.themodernlife" name := "simple-scalding-example" scalaVersion := "2.11.7" scalacOptions ++= Seq("-encoding", "utf-8", "-deprecation", "-unchecked", "-feature") resolvers ++= Seq( "Concurrent Maven Repo" at "http://conjars.org/repo", "Twitter Maven Repo" at "http://maven.twttr.com" ) libraryDependencies ++= Seq( "com.twitter" %% "scalding-core" % "0.15.0", "org.apache.hadoop" % "hadoop-client" % "2.2.0" % "provided", "org.slf4j" % "slf4j-log4j12" % "1.7.13" % "provided" ) A new Scalding build We need to update Scalding Apply https://github.com/twitter/scalding/pull/1446 Apply/hack https://github.com/twitter/scalding/pull/1220 Hack Build.scala to ignore some submodules According to http://www.cascading.org/2

## SoundCloud in Scalding case study by Concurrent Inc.

DevFeed: [SoundCloud in Scalding case study by Concurrent Inc.](<https://devfeed.tech/articles/soundcloud-in-scalding-case-study-by-concurrent-inc-2146.md>)

Original publisher: [Read original article](<https://developers.soundcloud.com/blog//soundcloud-concurrent-scalding-case-study>)

Published: 2014-12-02T00:00:00Z

Content type: article

Language: en

Sources: [SoundCloud Backstage Blog](<https://devfeed.tech/sources/soundcloud-backstage-blog.md>)

Topics: [Scalding](<https://devfeed.tech/topics/scalding.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>)

Tags: [case-study](<https://devfeed.tech/tags/case-study.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [data](<https://devfeed.tech/tags/data.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [posts](<https://devfeed.tech/tags/posts.md>), [scalding](<https://devfeed.tech/tags/scalding.md>), [search](<https://devfeed.tech/tags/search.md>), [spark](<https://devfeed.tech/tags/spark.md>)

### AI overview

SoundCloud describes a case study with Concurrent Inc. about using Scalding for data-driven products such as Search. Scalding supports rapid iteration, easy testing, and loose coupling between data-processing pipelines.

### Source excerpt

Recently we teamed up with Concurrent Inc., the backers of the data-processing framework Cascading, to do a case study of how we use...