# Getting started with Scalding

DevFeed: [Getting started with Scalding](<https://devfeed.tech/articles/getting-started-with-scalding-20845.md>)

Original publisher: [Read original article](<http://themodernlife.net/scala/hadoop/hdfs/sclading/2015/12/20/getting-started-with-scalding/>)

Published: 2015-12-20T08:15:13Z

Content type: tutorial

Language: en

Sources: [Ian Hummel](<https://devfeed.tech/sources/ian-hummel.md>)

Topics: [Scalding](<https://devfeed.tech/topics/scalding.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [Scala](<https://devfeed.tech/topics/scala.md>), [Testing](<https://devfeed.tech/topics/testing.md>), [Compression](<https://devfeed.tech/topics/compression.md>), [Logging](<https://devfeed.tech/topics/logging.md>)

Tags: [beginners](<https://devfeed.tech/tags/beginners.md>), [build](<https://devfeed.tech/tags/build.md>), [building](<https://devfeed.tech/tags/building.md>), [compression](<https://devfeed.tech/tags/compression.md>), [getting-started](<https://devfeed.tech/tags/getting-started.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [hdfs](<https://devfeed.tech/tags/hdfs.md>), [logging](<https://devfeed.tech/tags/logging.md>), [scala](<https://devfeed.tech/tags/scala.md>), [scalding](<https://devfeed.tech/tags/scalding.md>), [sclading](<https://devfeed.tech/tags/sclading.md>), [testing](<https://devfeed.tech/tags/testing.md>), [unit-testing](<https://devfeed.tech/tags/unit-testing.md>)

## AI overview

A practical guide to starting Scalding projects with SBT. It presents a minimal project structure and dependencies, explains local Hadoop execution and logging, and shows how to test jobs locally before submitting them to a cluster.

## Source excerpt

I've been using Scalding for the last few years and really love how simple it makes writing scalalbe data processing jobs. I think many of the issues beginners have with Scalding relate to project setup. I hope this post simplifies things for people so they can started with less hassle. Building your project with SBT The official getting started guide is pretty terse. For our purposes, all you need to get started is a very simple SBT project. simple-scalding-example/ |-- build.sbt |-- project | |-- build.properties | `-- plugins.sbt |-- src | `-- main | `-- scala | `-- WordCountJob.scala `-- log4j.properties The most important thing the setup of the build.sbt file. Our example has minimal dependencies: libraryDependencies ++= Seq( "com.twitter" %% "scalding-core" % "0.15.0", "org.apache.hadoop" % "hadoop-client" % "2.2.0" % "provided", "org.slf4j" % "slf4j-log4j12" % "1.7.13" % "provided" ) The hadoop-client jar is included for compilation, unit testing and running locally on your laptop but will not be pacakged into the final "fat jar" assembly shipped out to the cluster. Additionally, a logging framework is included so that we get good logs when debugging locally. A simple log4j.properties file is included in the repo. The whole job can be run locally using real Hadoop libraries without any additional software install: $ sbt 'run net.themodernlife.WordCount --hdfs --input build.sbt --output target/output' It's even possible to get things like LZO-compression to work with this setup. The main win here is that there is a very simple, repeatable development environment that can be used to test all aspects of the job locally before sending things off to the cluster.