# Running Apache Flink on Amazon Elastic Mapreduce

DevFeed: [Running Apache Flink on Amazon Elastic Mapreduce](<https://devfeed.tech/articles/running-apache-flink-on-amazon-elastic-mapreduce-20847.md>)

Original publisher: [Read original article](<http://themodernlife.net/scala/hadoop/hdfs/sclading/flink/streaming/realtime/emr/aws/2016/01/06/running-apache-flink-on-amazon-elastic-mapreduce/>)

Published: 2016-01-06T08:15:13Z

Content type: tutorial

Language: en

Sources: [Ian Hummel](<https://devfeed.tech/sources/ian-hummel.md>)

Topics: [apache-flink](<https://devfeed.tech/topics/apache-flink.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [Scalding](<https://devfeed.tech/topics/scalding.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>)

Tags: [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [aws](<https://devfeed.tech/tags/aws.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [ec2](<https://devfeed.tech/tags/ec2.md>), [emr](<https://devfeed.tech/tags/emr.md>), [flink](<https://devfeed.tech/tags/flink.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [hdfs](<https://devfeed.tech/tags/hdfs.md>), [linux](<https://devfeed.tech/tags/linux.md>), [realtime](<https://devfeed.tech/tags/realtime.md>), [s3](<https://devfeed.tech/tags/s3.md>), [scala](<https://devfeed.tech/tags/scala.md>), [scalding](<https://devfeed.tech/tags/scalding.md>), [sclading](<https://devfeed.tech/tags/sclading.md>), [spark](<https://devfeed.tech/tags/spark.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

## AI overview

A tutorial on running Apache Flink jobs on Amazon EMR, including adapting Scalding jobs and launching and accessing an EMR cluster. It also discusses EMRFS support for reading data from Amazon S3.

## Source excerpt

I love really Amazon EMR. Over the years it's grown from being "Hadoop on-demand" to a full-fledged cluster management system for running OSS big-data apps (Hadoop MR of course, but also Spark, Hue, Hive, Pig, Oozie and more). While Hadoop out of the box supports reading from S3, EMR has a proprietary implementation called EMRFS that has some nice features. For those reasons, it's really the best Hadoop cluster to use if you're storing your data in S3. Lately I've been experimenting a lot with Apache Flink to replace MR as the excution fabric. At work, we have many, many jobs written in Scalding. Flink can execute Scalding jobs with some very simple modifications which was a great way to move our jobs from MR to a more memory-centric data processing model. However, we really wanted to run our jobs on EMR using Flink. Flink is not an option for EMR (yet) but can we still get our jobs to run? Let's see! Start an EMR cluster The first thing we need is an EMR cluster. You can launch a small test cluster for very cheap. Once it's running, let's ssh onto it and see what's going on. ihummel at mm-mac-3270 in ~ $ ssh hadoop@ec2-54-226-25-85.compute-1.amazonaws.com Last login: Wed Jan 6 16:33:01 2016 from 172.85.47.138 __| __|_ ) _| ( / Amazon Linux AMI ___|\___|___| https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/ 23 package(s) needed for security, out of 49 available Run "sudo yum update" to apply all updates. EEEEEEEEEEEEEEEEEEEE MMMMMMMM MMMMMMMM RRRRRRRRRRRRRRR E::::::::::::::::::E M:::::::M M:::::::M R::::::::::::::R EE:::::EEEEEEEEE:::E M::::::::M M::::::::M R:::::RRRRRR:::::R E::::E EEEEE M:::::::::M M:::::::::M RR::::R R::::R E::::E M::::::M:::M M:::M::::::M R:::R R::::R E:::::EEEEEEEEEE M:::::M M:::M M:::M M:::::M R:::RRRRRR:::::R E::::::::::::::E M:::::M M:::M:::M M:::::M R:::::::::::RR E:::::EEEEEEEEEE M:::::M M:::::M M:::::M R:::RRRRRR::::R E::::E M:::::M M:::M M:::::M R:::R R::::R E::::E EEEEE M:::::M MMM M:::::M R:::R R::::R EE:::::EEEEEEEE::::E