# Running Scalding jobs on Apache Flink

DevFeed: [Running Scalding jobs on Apache Flink](<https://devfeed.tech/articles/running-scalding-jobs-on-apache-flink-20846.md>)

Original publisher: [Read original article](<http://themodernlife.net/scala/hadoop/hdfs/sclading/flink/streaming/realtime/2015/12/20/running-scalding-jobs-on-apache-flink/>)

Published: 2015-12-20T08:15:13Z

Content type: tutorial

Language: en

Sources: [Ian Hummel](<https://devfeed.tech/sources/ian-hummel.md>)

Topics: [Scalding](<https://devfeed.tech/topics/scalding.md>), [apache-flink](<https://devfeed.tech/topics/apache-flink.md>), [Scala](<https://devfeed.tech/topics/scala.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>)

Tags: [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [flink](<https://devfeed.tech/tags/flink.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [hdfs](<https://devfeed.tech/tags/hdfs.md>), [realtime](<https://devfeed.tech/tags/realtime.md>), [scala](<https://devfeed.tech/tags/scala.md>), [scalding](<https://devfeed.tech/tags/scalding.md>), [sclading](<https://devfeed.tech/tags/sclading.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

## AI overview

This tutorial explains how to run Scalding jobs on Apache Flink. It outlines the required compatibility work, including Cascading 3 support, a Scalding version compiled against Cascading 3, selection of hadoop2-flink as the Cascading fabric, and a fix for a Twitter Chill bug. It also provides a sample Scala word-count job and build configuration changes.

## Source excerpt

My previous post showed a very simple Scalding workflow. Apache Flink is a real time streaming framework that's very promising. It also supports running Cascading workflows with very little modification. Surely there must be some way to run a Scalding job on top of Flink? Turns out... YES! In a nutshell Here are the high-level things we need to solve for We need a Scalding job to test this out with cascading-flink requires Cascading 3 We need a new version of Scalding - Compiled against Cascading 3 - Allows hadoop2-flink to be selected as the Cascading "fabric" There is a bug in Twitter Chill We need Flink packaged with the Chill fix Scalding job Let's start with a very simple Scalding job. You can download https://github.com/themodernlife/simple-scalding-example for some inspiration. package net.themodernlife import com.twitter.scalding._ class WordCount(args: Args) extends Job(args) { def tokenize(text: String): Array[String] = { text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("\\s+") } val input = args("input") val output = args("output") TextLine(args("input")) .flatMap[String, String]('line -> 'word)(tokenize) .groupBy('word)(_.size) .write(Tsv(output)) } We'll be making some updates to our build.sbt. Here's what we're starting with organization := "net.themodernlife" name := "simple-scalding-example" scalaVersion := "2.11.7" scalacOptions ++= Seq("-encoding", "utf-8", "-deprecation", "-unchecked", "-feature") resolvers ++= Seq( "Concurrent Maven Repo" at "http://conjars.org/repo", "Twitter Maven Repo" at "http://maven.twttr.com" ) libraryDependencies ++= Seq( "com.twitter" %% "scalding-core" % "0.15.0", "org.apache.hadoop" % "hadoop-client" % "2.2.0" % "provided", "org.slf4j" % "slf4j-log4j12" % "1.7.13" % "provided" ) A new Scalding build We need to update Scalding Apply https://github.com/twitter/scalding/pull/1446 Apply/hack https://github.com/twitter/scalding/pull/1220 Hack Build.scala to ignore some submodules According to http://www.cascading.org/2