# Making Your Local Hadoop more like AWS Elastic MapReduce

DevFeed: [Making Your Local Hadoop more like AWS Elastic MapReduce](<https://devfeed.tech/articles/making-your-local-hadoop-more-like-aws-elastic-mapreduce-20844.md>)

Original publisher: [Read original article](<http://themodernlife.net/emr/hadoop/2014/01/02/making-local-hadoop-more-like-emr/>)

Published: 2014-01-02T11:21:13Z

Content type: tutorial

Language: en

Sources: [Ian Hummel](<https://devfeed.tech/sources/ian-hummel.md>)

Topics: [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [configuration](<https://devfeed.tech/topics/configuration.md>), [Homebrew](<https://devfeed.tech/topics/homebrew.md>), [Development](<https://devfeed.tech/topics/development.md>), [Package manager](<https://devfeed.tech/topics/package-manager.md>), [Compression](<https://devfeed.tech/topics/compression.md>), [Unix](<https://devfeed.tech/topics/unix.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [aws](<https://devfeed.tech/tags/aws.md>), [compression](<https://devfeed.tech/tags/compression.md>), [config](<https://devfeed.tech/tags/config.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [dev](<https://devfeed.tech/tags/dev.md>), [emr](<https://devfeed.tech/tags/emr.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [install](<https://devfeed.tech/tags/install.md>), [local](<https://devfeed.tech/tags/local.md>), [s3](<https://devfeed.tech/tags/s3.md>), [software-packages](<https://devfeed.tech/tags/software-packages.md>), [unix](<https://devfeed.tech/tags/unix.md>)

## AI overview

This tutorial explains how to configure a local Hadoop environment to more closely match AWS Elastic MapReduce. It covers using s3:// URIs, embedding AWS access keys, enabling transparent LZO compression, and installing Hadoop with Homebrew, including the distinction between s3:// and s3n:// handling in HDFS.

## Source excerpt

At MediaMath we're big users of Elastic MapReduce. EMR's incredible flexibility makes it a great fit for our analytics jobs. An extremely important best practice for any analytics project is to ensure your local dev and test environments match your production environment as much as possible. This eliminates the nasty surprise of launching a job that takes hours only to discover that it fails late into the run due to some unmet dependency or config mistake. Failing to invest time in the dev/test phase is a surefire way to blow big $$. Lately I've been investigating some configuration settings you can make to your local Hadoop to bring it inline with what you'll find when you run a job on an EMR cluster. This is especially important to us since we use S3 as a sort of centralized filesystem and EMR is designed to work wonderfully with S3. Specifically: Using s3:// URIs everywhere instead of s3n:// URIs Embedding AWS access keys Supporting transparent LZO compression Installing Hadoop I run all my Hadoop jobs on my laptop using Homebrew. Homebrew is a fantastic package manager for OS X that makes it a breeze to install general UNIX utilities as well as more complicated software packages (like Hadoop and Hive). $> brew install hadoop And you're good! s3:// vs s3n:// URIs in HDFS Ever wondered what the difference between an s3:// URI and an s3n:// URI is? Essentially up until December, 2010 S3 had a 5GB object size limit. So, if you used the default S3 HDFS implementation (by specifying an s3n:// URI) you couldn't read/write files greater than 5GB. That said, when you did read or write a file with HDFS there was a 1 to 1 correspondence with the object that got stored in S3. To process files larger than 5GB you had to use s3:// URIs in HDFS which actually chunked the file into multiple pieces behind the scenes before storing each piece as a separate object in S3. So when accessing something via HDFS with an s3://bucket/object URI you might actually be downloading multiple