# emr Published articles for emr. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Project Teleport: Cost-Effective and Scalable Kafka Data Processing at Block DevFeed: [Project Teleport: Cost-Effective and Scalable Kafka Data Processing at Block]() Original publisher: [Read original article]() Author: Unni Krishnan Published: 2025-03-20T00:00:00Z Content type: article Language: en Sources: [Cash App Code Blog]() Topics: [data-processing](), [Kafka](), [migration](), [databricks](), [data lake](), [Apache Spark](), [Amazon S3](), [parquet](), [Amazon Redshift]() Tags: [acquisition](), [amazon](), [cloud](), [data](), [data-lake](), [data-processing](), [databricks](), [delta-lake](), [emr](), [kafka](), [parquet](), [pipelines](), [redshift](), [s3](), [scale](), [storage]() ### AI overview Project Teleport is Block's cross-region Kafka data-processing system for integrating Afterpay's Sydney-hosted data lake into Block's US-based ecosystem. Built with Delta Lake, Spark on Databricks, and object storage, it supports migration of legacy pipelines and reduced cloud egress costs by USD 540,000 per year. ### Source excerpt Teleport achieves efficient and reliable cross-region Kafka data processing at scale. Using this approach, Afterpay data team reduced cloud egress costs by USD 540,000 per year. ## Stitch Fix's Move to Kubernetes for More Agile Service Deployment DevFeed: [Stitch Fix's Move to Kubernetes for More Agile Service Deployment]() Original publisher: [Read original article]() Published: 2023-09-19T15:00:00Z Content type: article Language: en Sources: [Stitch Fix]() Topics: [Deployment](), [Kubernetes](), [Amazon Elastic Kubernetes Service](), [Microservice](), [autoscaling](), [Apache Spark]() Tags: [autoscaling](), [deployment](), [emr](), [golang](), [kubernetes](), [microservices](), [nodejs](), [python](), [scala](), [spark]() ### AI overview This article describes Stitch Fix's effort to improve service deployment agility. It explains the platform's self-service goals, its transition from EMR-based Spark infrastructure to EKS, and the deployment challenges that motivated consideration of Kubernetes for core platform services, including standardization across languages, autoscaling, rollback, and traffic segmentation. ### Source excerpt Introduction At Stitch Fix, our data platform is designed to be self-service, with our users taking ownership of their own ETL, models, and microservices. To support this approach, the platform team prioritizes user autonomy and end-to-end tooling in their tooling designs, minimizing the involvement of platform engineers in day-to-day engineering and data science workflows. We regularly evaluate our infrastructure against new frameworks to assess the costs and benefits of potential updates. Last year, we transitioned our EMR-based Spark infrastructure to EKS to take advantage of its benefits. The next logical step was to re-examine how our core platform services were deployed and updated, especially those still deployed directly to instances. We also recognized that Kubernetes, a popular open-source system for handling the deployment of containerized applications, could provide benefits for microservice orchestration beyond just batch compute infrastructure. Our existing instance-based service deployment framework was beginning to show its age in areas that were critical for Stitch Fix, such as the development lifecycle's velocity. As we saw an opportunity to leverage Kubernetes to address these issues, we identified pain points in our service deployment ecosystem. Pain Points Building and deploying services can be a time-consuming process, involving multiple steps and dependencies. We needed a solution that would streamline this process and enable us to deploy services more efficiently. Polyglot environments, where multiple programming languages are used within the same system, can present a challenge for deployment standardization. At Stitch Fix, we use a variety of languages, including Python, Golang, Nodejs, and JVM (Java and Scala), making it difficult to establish consistent deployment practices across the board. We needed a way to deploy polyglot environments in a standardized and efficient manner. Autoscaling and rollback capabilities are essential for manag ## How to Setup a Scheduled Scala Spark Job DevFeed: [How to Setup a Scheduled Scala Spark Job]() Original publisher: [Read original article]() Published: 2019-03-27T00:00:00Z Content type: tutorial Language: en Sources: [Curalate]() Topics: [Apache Spark](), [Scala](), [Amazon Web Services](), [Amazon S3](), [Hadoop](), [SDKs](), [Command-line interface]() Tags: [aws](), [cli](), [daily](), [data](), [ec2](), [emr](), [github](), [hadoop](), [how-to](), [job](), [pipeline](), [s3](), [scala](), [scheduled](), [sdk](), [spark](), [weekly]() ### AI overview A tutorial for packaging and deploying a Scala Spark job as a fat JAR, uploading it to Amazon S3, and configuring it to run on a schedule through AWS Data Pipeline and EMR. ### Source excerpt Have you written a Scala Spark job that processes a massive amount of data on an intimidating amount of RAM and you want to run it daily/weekly/monthly on a schedule on AWS? I had to do this recently, and couldn't find a good tutorial on the full process to get the spark job running. Included in this article and accompanying repository is everything you need to get your Scala Spark job running on AWS Data Pipeline and EMR. Code Repo This tutorial is not going to walk you through the process of actually writing your specific Scala Spark job to do whatever number crunching you need. There are already plenty of resources available (1, 2, 3) to get you started on that. The code template for setting up a Spark Scala job is available in this GitHub repo. Assuming that you have already written your Spark Job and are only using the AWS Java SDK to connect to your AWS data stores, drop your code in the Main function of SparkJob.scala and run the deploy.sh script to upload the fat jar to your S3 bucket. If you do take other dependencies, then it may take some extra work on your part. To run a Scala Spark job on AWS you need to compile a fat jar that contains the byte code for your job and all of the libraries it needs to run. This project already has the sbt-assembly plugin setup and a assemblyMergeStrategy set up to package the Spark, Hadoop, and AWS SDK together in the fat jar. If you need to add in other libraries that do not play well with each other, or are using a noncompatible version of Spark for this current repo, there are a few good resources available to help you through the needed build.sbt modifications. Outside of the previously mentioned needed changes you need to set a few parameters in the deploy.sh script. Mainly the deploymentPath to your specific S3 bucket, adding a profile to the AWS CLI command to upload to your specific S3 bucket if it's private, and changing the resulting fat jar name if you please. The deploy script uses the AWS CLI to upload the fat ## Running Apache Flink on Amazon Elastic Mapreduce DevFeed: [Running Apache Flink on Amazon Elastic Mapreduce]() Original publisher: [Read original article]() Published: 2016-01-06T08:15:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [apache-flink](), [Amazon Web Services](), [Hadoop](), [Scalding](), [data-processing](), [Amazon S3](), [Apache Spark]() Tags: [apache-flink](), [aws](), [data-processing](), [ec2](), [emr](), [flink](), [hadoop](), [hdfs](), [linux](), [realtime](), [s3](), [scala](), [scalding](), [sclading](), [spark](), [streaming]() ### AI overview A tutorial on running Apache Flink jobs on Amazon EMR, including adapting Scalding jobs and launching and accessing an EMR cluster. It also discusses EMRFS support for reading data from Amazon S3. ### Source excerpt I love really Amazon EMR. Over the years it's grown from being "Hadoop on-demand" to a full-fledged cluster management system for running OSS big-data apps (Hadoop MR of course, but also Spark, Hue, Hive, Pig, Oozie and more). While Hadoop out of the box supports reading from S3, EMR has a proprietary implementation called EMRFS that has some nice features. For those reasons, it's really the best Hadoop cluster to use if you're storing your data in S3. Lately I've been experimenting a lot with Apache Flink to replace MR as the excution fabric. At work, we have many, many jobs written in Scalding. Flink can execute Scalding jobs with some very simple modifications which was a great way to move our jobs from MR to a more memory-centric data processing model. However, we really wanted to run our jobs on EMR using Flink. Flink is not an option for EMR (yet) but can we still get our jobs to run? Let's see! Start an EMR cluster The first thing we need is an EMR cluster. You can launch a small test cluster for very cheap. Once it's running, let's ssh onto it and see what's going on. ihummel at mm-mac-3270 in ~ $ ssh hadoop@ec2-54-226-25-85.compute-1.amazonaws.com Last login: Wed Jan 6 16:33:01 2016 from 172.85.47.138 __| __|_ ) _| ( / Amazon Linux AMI ___|\___|___| https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/ 23 package(s) needed for security, out of 49 available Run "sudo yum update" to apply all updates. EEEEEEEEEEEEEEEEEEEE MMMMMMMM MMMMMMMM RRRRRRRRRRRRRRR E::::::::::::::::::E M:::::::M M:::::::M R::::::::::::::R EE:::::EEEEEEEEE:::E M::::::::M M::::::::M R:::::RRRRRR:::::R E::::E EEEEE M:::::::::M M:::::::::M RR::::R R::::R E::::E M::::::M:::M M:::M::::::M R:::R R::::R E:::::EEEEEEEEEE M:::::M M:::M M:::M M:::::M R:::RRRRRR:::::R E::::::::::::::E M:::::M M:::M:::M M:::::M R:::::::::::RR E:::::EEEEEEEEEE M:::::M M:::::M M:::::M R:::RRRRRR::::R E::::E M:::::M M:::M M:::::M R:::R R::::R E::::E EEEEE M:::::M MMM M:::::M R:::R R::::R EE:::::EEEEEEEE::::E ## Making Your Local Hadoop more like AWS Elastic MapReduce DevFeed: [Making Your Local Hadoop more like AWS Elastic MapReduce]() Original publisher: [Read original article]() Published: 2014-01-02T11:21:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [Hadoop](), [Amazon S3](), [configuration](), [Homebrew](), [Development](), [Package manager](), [Compression](), [Unix]() Tags: [analytics](), [aws](), [compression](), [config](), [configuration](), [dev](), [emr](), [hadoop](), [install](), [local](), [s3](), [software-packages](), [unix]() ### AI overview This tutorial explains how to configure a local Hadoop environment to more closely match AWS Elastic MapReduce. It covers using s3:// URIs, embedding AWS access keys, enabling transparent LZO compression, and installing Hadoop with Homebrew, including the distinction between s3:// and s3n:// handling in HDFS. ### Source excerpt At MediaMath we're big users of Elastic MapReduce. EMR's incredible flexibility makes it a great fit for our analytics jobs. An extremely important best practice for any analytics project is to ensure your local dev and test environments match your production environment as much as possible. This eliminates the nasty surprise of launching a job that takes hours only to discover that it fails late into the run due to some unmet dependency or config mistake. Failing to invest time in the dev/test phase is a surefire way to blow big $$. Lately I've been investigating some configuration settings you can make to your local Hadoop to bring it inline with what you'll find when you run a job on an EMR cluster. This is especially important to us since we use S3 as a sort of centralized filesystem and EMR is designed to work wonderfully with S3. Specifically: Using s3:// URIs everywhere instead of s3n:// URIs Embedding AWS access keys Supporting transparent LZO compression Installing Hadoop I run all my Hadoop jobs on my laptop using Homebrew. Homebrew is a fantastic package manager for OS X that makes it a breeze to install general UNIX utilities as well as more complicated software packages (like Hadoop and Hive). $> brew install hadoop And you're good! s3:// vs s3n:// URIs in HDFS Ever wondered what the difference between an s3:// URI and an s3n:// URI is? Essentially up until December, 2010 S3 had a 5GB object size limit. So, if you used the default S3 HDFS implementation (by specifying an s3n:// URI) you couldn't read/write files greater than 5GB. That said, when you did read or write a file with HDFS there was a 1 to 1 correspondence with the object that got stored in S3. To process files larger than 5GB you had to use s3:// URIs in HDFS which actually chunked the file into multiple pieces behind the scenes before storing each piece as a separate object in S3. So when accessing something via HDFS with an s3://bucket/object URI you might actually be downloading multiple