# Ian Hummel description goes here This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Running Apache Flink on Amazon Elastic Mapreduce DevFeed: [Running Apache Flink on Amazon Elastic Mapreduce]() Original publisher: [Read original article]() Published: 2016-01-06T08:15:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [apache-flink](), [Amazon Web Services](), [Hadoop](), [Scalding](), [data-processing](), [Amazon S3](), [Apache Spark]() Tags: [apache-flink](), [aws](), [data-processing](), [ec2](), [emr](), [flink](), [hadoop](), [hdfs](), [linux](), [realtime](), [s3](), [scala](), [scalding](), [sclading](), [spark](), [streaming]() ### AI overview A tutorial on running Apache Flink jobs on Amazon EMR, including adapting Scalding jobs and launching and accessing an EMR cluster. It also discusses EMRFS support for reading data from Amazon S3. ### Source excerpt I love really Amazon EMR. Over the years it's grown from being "Hadoop on-demand" to a full-fledged cluster management system for running OSS big-data apps (Hadoop MR of course, but also Spark, Hue, Hive, Pig, Oozie and more). While Hadoop out of the box supports reading from S3, EMR has a proprietary implementation called EMRFS that has some nice features. For those reasons, it's really the best Hadoop cluster to use if you're storing your data in S3. Lately I've been experimenting a lot with Apache Flink to replace MR as the excution fabric. At work, we have many, many jobs written in Scalding. Flink can execute Scalding jobs with some very simple modifications which was a great way to move our jobs from MR to a more memory-centric data processing model. However, we really wanted to run our jobs on EMR using Flink. Flink is not an option for EMR (yet) but can we still get our jobs to run? Let's see! Start an EMR cluster The first thing we need is an EMR cluster. You can launch a small test cluster for very cheap. Once it's running, let's ssh onto it and see what's going on. ihummel at mm-mac-3270 in ~ $ ssh hadoop@ec2-54-226-25-85.compute-1.amazonaws.com Last login: Wed Jan 6 16:33:01 2016 from 172.85.47.138 __| __|_ ) _| ( / Amazon Linux AMI ___|\___|___| https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/ 23 package(s) needed for security, out of 49 available Run "sudo yum update" to apply all updates. EEEEEEEEEEEEEEEEEEEE MMMMMMMM MMMMMMMM RRRRRRRRRRRRRRR E::::::::::::::::::E M:::::::M M:::::::M R::::::::::::::R EE:::::EEEEEEEEE:::E M::::::::M M::::::::M R:::::RRRRRR:::::R E::::E EEEEE M:::::::::M M:::::::::M RR::::R R::::R E::::E M::::::M:::M M:::M::::::M R:::R R::::R E:::::EEEEEEEEEE M:::::M M:::M M:::M M:::::M R:::RRRRRR:::::R E::::::::::::::E M:::::M M:::M:::M M:::::M R:::::::::::RR E:::::EEEEEEEEEE M:::::M M:::::M M:::::M R:::RRRRRR::::R E::::E M:::::M M:::M M:::::M R:::R R::::R E::::E EEEEE M:::::M MMM M:::::M R:::R R::::R EE:::::EEEEEEEE::::E ## Getting started with Scalding DevFeed: [Getting started with Scalding]() Original publisher: [Read original article]() Published: 2015-12-20T08:15:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [Scalding](), [data-processing](), [Hadoop](), [Scala](), [Testing](), [Compression](), [Logging]() Tags: [beginners](), [build](), [building](), [compression](), [getting-started](), [hadoop](), [hdfs](), [logging](), [scala](), [scalding](), [sclading](), [testing](), [unit-testing]() ### AI overview A practical guide to starting Scalding projects with SBT. It presents a minimal project structure and dependencies, explains local Hadoop execution and logging, and shows how to test jobs locally before submitting them to a cluster. ### Source excerpt I've been using Scalding for the last few years and really love how simple it makes writing scalalbe data processing jobs. I think many of the issues beginners have with Scalding relate to project setup. I hope this post simplifies things for people so they can started with less hassle. Building your project with SBT The official getting started guide is pretty terse. For our purposes, all you need to get started is a very simple SBT project. simple-scalding-example/ |-- build.sbt |-- project | |-- build.properties | `-- plugins.sbt |-- src | `-- main | `-- scala | `-- WordCountJob.scala `-- log4j.properties The most important thing the setup of the build.sbt file. Our example has minimal dependencies: libraryDependencies ++= Seq( "com.twitter" %% "scalding-core" % "0.15.0", "org.apache.hadoop" % "hadoop-client" % "2.2.0" % "provided", "org.slf4j" % "slf4j-log4j12" % "1.7.13" % "provided" ) The hadoop-client jar is included for compilation, unit testing and running locally on your laptop but will not be pacakged into the final "fat jar" assembly shipped out to the cluster. Additionally, a logging framework is included so that we get good logs when debugging locally. A simple log4j.properties file is included in the repo. The whole job can be run locally using real Hadoop libraries without any additional software install: $ sbt 'run net.themodernlife.WordCount --hdfs --input build.sbt --output target/output' It's even possible to get things like LZO-compression to work with this setup. The main win here is that there is a very simple, repeatable development environment that can be used to test all aspects of the job locally before sending things off to the cluster. ## Running Scalding jobs on Apache Flink DevFeed: [Running Scalding jobs on Apache Flink]() Original publisher: [Read original article]() Published: 2015-12-20T08:15:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [Scalding](), [apache-flink](), [Scala](), [Streaming](), [Hadoop]() Tags: [apache-flink](), [flink](), [hadoop](), [hdfs](), [realtime](), [scala](), [scalding](), [sclading](), [streaming]() ### AI overview This tutorial explains how to run Scalding jobs on Apache Flink. It outlines the required compatibility work, including Cascading 3 support, a Scalding version compiled against Cascading 3, selection of hadoop2-flink as the Cascading fabric, and a fix for a Twitter Chill bug. It also provides a sample Scala word-count job and build configuration changes. ### Source excerpt My previous post showed a very simple Scalding workflow. Apache Flink is a real time streaming framework that's very promising. It also supports running Cascading workflows with very little modification. Surely there must be some way to run a Scalding job on top of Flink? Turns out... YES! In a nutshell Here are the high-level things we need to solve for We need a Scalding job to test this out with cascading-flink requires Cascading 3 We need a new version of Scalding - Compiled against Cascading 3 - Allows hadoop2-flink to be selected as the Cascading "fabric" There is a bug in Twitter Chill We need Flink packaged with the Chill fix Scalding job Let's start with a very simple Scalding job. You can download https://github.com/themodernlife/simple-scalding-example for some inspiration. package net.themodernlife import com.twitter.scalding._ class WordCount(args: Args) extends Job(args) { def tokenize(text: String): Array[String] = { text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("\\s+") } val input = args("input") val output = args("output") TextLine(args("input")) .flatMap[String, String]('line -> 'word)(tokenize) .groupBy('word)(_.size) .write(Tsv(output)) } We'll be making some updates to our build.sbt. Here's what we're starting with organization := "net.themodernlife" name := "simple-scalding-example" scalaVersion := "2.11.7" scalacOptions ++= Seq("-encoding", "utf-8", "-deprecation", "-unchecked", "-feature") resolvers ++= Seq( "Concurrent Maven Repo" at "http://conjars.org/repo", "Twitter Maven Repo" at "http://maven.twttr.com" ) libraryDependencies ++= Seq( "com.twitter" %% "scalding-core" % "0.15.0", "org.apache.hadoop" % "hadoop-client" % "2.2.0" % "provided", "org.slf4j" % "slf4j-log4j12" % "1.7.13" % "provided" ) A new Scalding build We need to update Scalding Apply https://github.com/twitter/scalding/pull/1446 Apply/hack https://github.com/twitter/scalding/pull/1220 Hack Build.scala to ignore some submodules According to http://www.cascading.org/2 ## Getting the current filename with Spark and HDFS DevFeed: [Getting the current filename with Spark and HDFS]() Original publisher: [Read original article]() Published: 2014-09-28T11:50:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [Apache Spark](), [Hadoop]() Tags: [apache-spark](), [foreach](), [hadoop](), [hdfs](), [map](), [reduce](), [scala](), [spark](), [subclass]() ### AI overview A tutorial shows how to obtain the filename being processed in a Spark map/reduce job. It uses Spark 1.1.0's HadoopRDD.mapPartitionsWithInputSplit with FileSplit, and notes that the approach requires an InputFormat subclass of FileInputFormat. ### Source excerpt It's occasionally useful when writing map/reduce jobs to get a hold of the current filename that's being processed. There's a few ways to do this, depending on the version of Spark that you're using. Spark 1.1.0 introduced a new method on HadoopRDD that makes this super easy: import org.apache.hadoop.io.LongWritable import org.apache.hadoop.io.Text import org.apache.hadoop.mapred.{FileSplit, TextInputFormat} import org.apache.spark.rdd.HadoopRDD // Create the text file val text = sc.hadoopFile("file:///tmp/files", classOf[TextInputFormat], classOf[LongWritable], classOf[Text], sc.defaultMinPartitions) // text: org.apache.spark.rdd.RDD[(org.apache.hadoop.io.LongWritable, org.apache.hadoop.io.Text)] = file:///tmp/files HadoopRDD[2] at hadoopFile at :16 // Cast to a HadoopRDD val hadoopRdd = text.asInstanceOf[HadoopRDD[LongWritable, Text]] // hadoopRdd: org.apache.spark.rdd.HadoopRDD[org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text] = file:///tmp/files HadoopRDD[2] at hadoopFile at :16 val fileAndLine = hadoopRdd.mapPartitionsWithInputSplit { (inputSplit, iterator) ⇒ val file = inputSplit.asInstanceOf[FileSplit] iterator.map { tpl ⇒ (file.getPath, tpl._2) } } fileAndLine.foreach(println) // prints lines like (file:/tmp/files/part-00000,line1...), (file:/tmp/files/part-00000,line2...), ... There are a few things to note: - You can't simply use sc.textFile because under the hood, the textFile method actually maps the resulting HadoopRDD, returning a MappedRDD which is not what we want - This will only work if your InputFormat is a subclass of FileInputFormat - mapPartitionsWithInputSplit returns a new iterator, in this case we simply map over it (which is done lazily!) to include the split's filename ## Using Play validation with Scala for strongly typed Hadoop and Spark data pipelines DevFeed: [Using Play validation with Scala for strongly typed Hadoop and Spark data pipelines]() Original publisher: [Read original article]() Published: 2014-08-07T11:50:13Z Content type: article Language: en Sources: [Ian Hummel]() Topics: [Scala](), [data-processing](), [Apache Spark](), [Hadoop](), [Back end](), [Exception]() Tags: [backend](), [big-data](), [data-processing](), [framework](), [hadoop](), [play](), [scala](), [spark](), [validation]() ### AI overview The article examines whether Play's validation API, commonly used for HTML form and JSON submissions, can also support backend data pipelines powered by Hadoop or Spark. It describes applying validation during translation of encoded records into strongly typed Scala data. ### Source excerpt The other day I was talking with a colleague about data validation and the Play web framework came up. Play has a nice API for validating HTML form and JSON submissions. This works great when you're processing small amounts of data from the web-tier of your application. But could that same tech benefit a Big Data team working on a backend powered by Hadoop or Spark? We decided to find out and the results were encouraging. The secret sauce? Play's combinator-based approach to data validation. Whether your data is big or small, garbage in is garbage out MediaMath processes TBs of online user behavior and advertising data every day. It's inevitable that with hundreds of machines spread across multiple datacenters, legacy systems and partner provided APIs we receive bad data or invalid records from time to time. Systems built around file formats like CSV or TSV are especially susceptible to encoding errors that can cause headaches for downstream processing systems. So what are your options? Quite often the first step for most data processing pipelines (be they single node scripts or massive Hadoop jobs) is translating some kind of encoded wire format T into a record of type D for partitioning, joining, filtering or aggregating. In mathematical terms, you need a function translate: (input: T) => D where input could be a parsed JSON object, a snippet of XML, an array of bytes or an array of stings in the case of tab or comma delimited files. But what if the translation fails? Think about the scenario of processing a CSV file line by line. Each line has columns of different types (strings, integers, floating point numbers). What if someone puts "#@$?" where you were expecting a number? Or leaves a required field blank? In other words, our function is only defined for some values of T (it's a partial function). At MediaMath we use Scala, so the natural choice would be to model this by throwing an exception or returning an Option[D]. That said, a richer validation API would ## Scala Unicode Arrows in IntelliJ IDEA DevFeed: [Scala Unicode Arrows in IntelliJ IDEA]() Original publisher: [Read original article]() Published: 2014-01-08T11:50:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [IntelliJ IDEA](), [Scala](), [code-completion](), [Sublime Text]() Tags: [code-completion](), [coding](), [github](), [idea](), [intellij](), [intellij-idea](), [scala]() ### AI overview A tutorial explains how to use IntelliJ IDEA Live Templates to convert typed Scala operators such as =>, ->, and <- into Unicode arrows. It also contrasts this workflow with a Sublime Text project that provides Scala tab completion. ### Source excerpt Several of my colleagues love IntelliJ for coding in Scala. I was pretty happy with Sublime Text 2 (and still use it for Ruby/Python/Shell/whatever) but the lack of code completion was really starting to affect my productivity. I spent way too much time looping through the edit/compile/fix typo cycle. Before I could switch though, I really wanted my fancy arrows in Scala! I have a GitHub project which adds tab completion in Sublime Text for Scala to turn "=>" into "⇒", "->" into "->" and "<-" into "<-". Turns out this is really easy to accomplish in IntelliJ as well! You just need to create a few Live Templates The Abbreviation field should be set to "=>" and the Template text field should be "⇒ " (I left a trailing space so my cursor position gets updated). Repeat for any other operators. Also ensure that you change the applicable context to Scala. Now when you type "=>" and hit you should get a unicode arrow, "⇒"! ## Making Your Local Hadoop more like AWS Elastic MapReduce DevFeed: [Making Your Local Hadoop more like AWS Elastic MapReduce]() Original publisher: [Read original article]() Published: 2014-01-02T11:21:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [Hadoop](), [Amazon S3](), [configuration](), [Homebrew](), [Development](), [Package manager](), [Compression](), [Unix]() Tags: [analytics](), [aws](), [compression](), [config](), [configuration](), [dev](), [emr](), [hadoop](), [install](), [local](), [s3](), [software-packages](), [unix]() ### AI overview This tutorial explains how to configure a local Hadoop environment to more closely match AWS Elastic MapReduce. It covers using s3:// URIs, embedding AWS access keys, enabling transparent LZO compression, and installing Hadoop with Homebrew, including the distinction between s3:// and s3n:// handling in HDFS. ### Source excerpt At MediaMath we're big users of Elastic MapReduce. EMR's incredible flexibility makes it a great fit for our analytics jobs. An extremely important best practice for any analytics project is to ensure your local dev and test environments match your production environment as much as possible. This eliminates the nasty surprise of launching a job that takes hours only to discover that it fails late into the run due to some unmet dependency or config mistake. Failing to invest time in the dev/test phase is a surefire way to blow big $$. Lately I've been investigating some configuration settings you can make to your local Hadoop to bring it inline with what you'll find when you run a job on an EMR cluster. This is especially important to us since we use S3 as a sort of centralized filesystem and EMR is designed to work wonderfully with S3. Specifically: Using s3:// URIs everywhere instead of s3n:// URIs Embedding AWS access keys Supporting transparent LZO compression Installing Hadoop I run all my Hadoop jobs on my laptop using Homebrew. Homebrew is a fantastic package manager for OS X that makes it a breeze to install general UNIX utilities as well as more complicated software packages (like Hadoop and Hive). $> brew install hadoop And you're good! s3:// vs s3n:// URIs in HDFS Ever wondered what the difference between an s3:// URI and an s3n:// URI is? Essentially up until December, 2010 S3 had a 5GB object size limit. So, if you used the default S3 HDFS implementation (by specifying an s3n:// URI) you couldn't read/write files greater than 5GB. That said, when you did read or write a file with HDFS there was a 1 to 1 correspondence with the object that got stored in S3. To process files larger than 5GB you had to use s3:// URIs in HDFS which actually chunked the file into multiple pieces behind the scenes before storing each piece as a separate object in S3. So when accessing something via HDFS with an s3://bucket/object URI you might actually be downloading multiple