# Hadoop Apache Hadoop is an open-source software framework for distributed processing of large data sets across computer clusters. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering DevFeed: [The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-08T12:16:09Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [pandas](), [data-engineering](), [Artificial Intelligence](), [DuckDB](), [data-platforms](), [software-development](), [future of software](), [Apache Spark](), [Hadoop](), [Open Source](), [parquet]() Tags: [ai](), [apache-arrow](), [arrow](), [data](), [data-engineering](), [data-platforms](), [duckdb](), [hadoop](), [large-language-models](), [open-source](), [pandas](), [parquet](), [software-development]() ### AI overview An interview with Wes McKinney covers the origins of pandas and Apache Arrow, the evolution of modern data engineering from Hadoop to lakehouse architectures, and the roles of tools such as Parquet, DuckDB, DataFusion, and Spark. McKinney also discusses how AI affects software development, arguing that it can improve experienced engineers' productivity but does not replace software engineering, architecture, or judgment. ### Source excerpt interview with Wes McKinney ## Booking.com 2026 GenAI and ML PhD Research Internship in Amsterdam DevFeed: [Booking.com 2026 GenAI and ML PhD Research Internship in Amsterdam]() Original publisher: [Read original article]() Author: Yang Yang Published: 2026-02-05T10:39:26Z Content type: article Language: en Sources: [Booking.com Data Science]() Topics: [Machine Learning & Artificial Intelligence](), [genai](), [AI Research](), [Python](), [Large Language Model](), [Apache Spark](), [SQL](), [Hadoop](), [Reinforcement learning](), [Synthetic Data Generation](), [Transformer]() Tags: [2026](), [ai](), [big-data](), [blog-posts](), [featured](), [genai](), [internship](), [machine-learning](), [ml](), [python](), [research]() ### AI overview Booking.com is recruiting current PhD students in quantitative fields for a three-month GenAI and machine learning research internship in Amsterdam in 2026. Projects include LLM alignment, transformer explainability, embeddings, context engineering, and synthetic data generation. ### Source excerpt At Booking.com, we don't just use Machine Learning -- we use it to solve some of the most complex travel challenges in the world. We're looking for the next generation of researchers to join our Machine Learning community in Amsterdam for a 3-month deep dive into cutting-edge AI. The Program As a Research Intern, you'll be embedded in our teams, working alongside world-class mentors. Your mission? To tackle real-world problems and push the boundaries of the state-of-the-art. Are You the One? We're looking for current PhD students in quantitative fields (CS, Math, AI, Physics) who can conduct independent research and have a solid grip on Python and Big Data tech (SQL, Spark, Hadoop). What's in it for you? You won't just be "an intern". You'll be a contributor to our Machine Learning community. You'll have the opportunity to contribute to the existing efforts of the Machine Learning teams, participate in internal knowledge-sharing sessions, and enjoy the collaborative, high-energy environment of our Amsterdam HQ. Projects Regularized Target Encoding for large real-world datasets Multi-Agent Collaboration Aligning LLMs with user feedback via reinforcement learning Multi-level treatments Interpretable Foundations: Explainability Methods for Transformer Models on Sequential Event Data Scalable and generalisable ID embedding learning Improving property embeddings with better handling of rich and long-context data Utility-aware retrieval for context engineering in travel planning Synthetic Data Generation in Images Requirements We are looking for independent researchers with strong understanding of Machine Learning topics (see requirements for each project in the Linkedin ad), have a track record of peer-reviewed publications and a passion for solving complex problems. Why Booking.com? You'll join a vibrant, diverse community of data scientists and researchers who love to experiment. Beyond the code, you'll experience the unique culture of our Amsterdam headquarters -- a hub ## Out with the old file system DevFeed: [Out with the old file system]() Original publisher: [Read original article]() Author: Manfred Moser, David Phillips, Mateusz Gajewski Published: 2025-02-10T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Filesystems](), [Apache Iceberg](), [Hadoop](), [Amazon S3](), [Azure](), [Google Cloud Platform (GCP)]() Tags: [amazon-s3](), [apache-iceberg](), [apache-parquet](), [azure](), [compression](), [data-lake](), [deprecated](), [google-cloud](), [hadoop](), [maintenance](), [release]() ### AI overview Trino 470 deprecated its legacy Hadoop-based file system support, which will be removed in a future release. The article explains Trino's move toward custom file system implementations for cloud storage and describes the migration path for users, including catalog-level file system configuration and warnings for deprecated properties. ### Source excerpt What a long journey it has been! From the start Trino supported querying Hive data and used libraries from the Hive and Hadoop ecosystem. With the release of Trino 470 we mark another milestone to more features and better performance for data lake and lakehouse querying with Trino. We deprecated the legacy file system support, and will permanently remove them in an upcoming release. ## How Uber Reduced Their Log Size By 99% DevFeed: [How Uber Reduced Their Log Size By 99%]() Original publisher: [Read original article]() Author: Richard Oliver Bray Published: 2024-10-09T13:02:55Z Content type: article Language: en Sources: [Hacking Scale by Better Stack]() Topics: [Logging](), [Hadoop](), [Filesystems](), [big-data](), [Apache Spark](), [data]() Tags: [apache-spark](), [big-data](), [cli](), [clickhouse](), [data](), [hadoop](), [hdfs](), [logging]() ### AI overview The article explains how Uber addressed the storage cost of generating roughly 5 PB of INFO-level logs each month. It describes Uber's use of HDFS and related data-processing tools, while reporting that the company reduced log storage size by 99%. ### Source excerpt Uber broke apart an open source tool to massively compress their logs ## Implementing Data Validation with Great Expectations in Hybrid Environments DevFeed: [Implementing Data Validation with Great Expectations in Hybrid Environments]() Original publisher: [Read original article]() Author: Kamila Widyanto Full time DevOps; Site Reliability Engineer; Part Time Rubberduck Linkedin Profile Published: 2023-04-25T00:00:00Z Content type: tutorial Language: en Sources: [Trivago]() Topics: [data-processing](), [Hadoop](), [hdfs](), [integrity](), [Apache Spark](), [configuration](), [Library](), [Python](), [JSON](), [Open Source](), [YAML](), [version-control]() Tags: [configuration](), [data](), [data-pipeline](), [data-processing](), [data-validation](), [devops](), [distributed](), [hadoop](), [hdfs](), [integrity](), [json](), [library](), [open-source](), [python](), [version-control](), [workflow](), [yaml]() ### AI overview This article describes implementing Great Expectations for data validation in a hybrid Hadoop environment. It explains the framework's core concepts and how the authors ran it as a PySpark job in an automated data pipeline, including configuring the Data Context for HDFS constraints. ### Source excerpt Data validation is an essential step in any data processing pipeline, as it ensures the integrity and accuracy of the data to be used across all subsequent processing steps. ## Journey to Iceberg with Trino DevFeed: [Journey to Iceberg with Trino]() Original publisher: [Read original article]() Author: JaeChang Song, Jennifer Oh, Brian Olsen Published: 2022-12-19T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Apache Iceberg](), [data-engineering](), [big-data](), [dashboards](), [Hadoop]() Tags: [data-engineering](), [engineering](), [hadoop](), [hdfs](), [logs](), [metrics](), [open-source](), [performance](), [post](), [scale](), [speed](), [summit](), [switching]() ### AI overview SK Telecom describes its journey from Hive-based Trino deployments to Iceberg after encountering scaling and performance problems. The company used Trino across Hadoop and HDFS-based data platforms, collected query plans, JMX statistics, system metrics, and logs, and built a dashboard to investigate blocked queries and cluster behavior. ### Source excerpt This post comes from the second half of Trino Summit 2022 session. Our friends JaeChang and Jennifer from SK Telecom traveled across the globe from South Korea to join us in person! SK Telecom recently had some issues scaling Trino on the Hive model, among other issues that come with Hive. While some initial tweaking helped speed things up, it ultimately never solved the problem. After switching to Iceberg, SK Telecom ran initial performance tests with some very impressive results. In this talk, Jennifer and JaeChang describe their journey to Iceberg with Trino. ## Обзор плагина Big Data Tools DevFeed: [Обзор плагина Big Data Tools]() Original publisher: [Read original article]() Author: olegchir (JetBrains) Published: 2021-07-28T10:41:36Z Content type: article Language: ru Sources: [JetBrains RU]() Topics: [ide](), [big-data](), [Hadoop](), [Apache Spark](), [data](), [Scala](), [Python]() Tags: [big-data](), [big-data-tools](), [data](), [hadoop](), [ide](), [jetbrains](), [kafka](), [python](), [scala](), [spark](), [tools](), [zeppelin]() ### AI overview This article reviews JetBrains' Big Data Tools plugin for working with cloud file systems, Hadoop, Spark, and Zeppelin directly from an IDE. It explains the plugin's role in data-engineering workflows, including ETL, and notes support for Scala and Python. ### Source excerpt Храните файлы в облачных файловых системах или, может быть, используете Hadoop, Spark и Zeppelin? А пробовали ли вы работать с ними напрямую из IDE? Привет, меня зовут Олег, я из команды плагина Big Data Tools. В этой статье мы поговорим, зачем этот плагин нужен, как применяется и где его достать. За последний год плагин прошёл большой путь и из экспериментального продукта превратился в боевое решение, на которое стоит взглянуть специалистам по Big Data. В JetBrains мы создаем IDE и другие инструменты, которые делают жизнь разработчиков лучше. Big Data Tools -- это очень узкоспециализированный, редкоземельный плагин, который предназначен для конкретного вида разработчиков -- для дата-инженеров. Если вам интересно подробней узнать о мире Big Data и работе дата-инженеров, рекомендую развернутую серию статей Паши Финкельштейна. Здесь мы рассмотрим одну из самых популярных схем. Читать далее ## We're rebranding PrestoSQL as Trino DevFeed: [We're rebranding PrestoSQL as Trino]() Original publisher: [Read original article]() Author: Martin Traverso, Dain Sundstrom, David Phillips Published: 2020-12-27T00:00:00Z Content type: release Language: en Sources: [Trino Blog]() Topics: [Software](), [Open Source](), [Data Infrastructure](), [Hadoop](), [GitHub](), [Slack]() Tags: [community](), [contributors](), [data-infrastructure](), [github](), [hadoop](), [infrastructure](), [open-source](), [rebranding]() ### AI overview The article announces that PrestoSQL is being rebranded as Trino. It explains that the software and community remain intact while describing the project's origins in low-latency analytics over Hadoop data and its commitment to an open, independent, collaborative community. ### Source excerpt We're rebranding PrestoSQL as Trino. The software and the community you have come to love and depend on aren't going anywhere, we are simply renaming. Trino is the new name for PrestoSQL, the project supported by the founders and creators of Presto® along with the major contributors - just under a shiny new name. And now you can find us here: GitHub: https://github.com/trinodb/trino. Please give it a star! Twitter: @trinodb Slack: https://trino.io/slack.html If you want to learn why we're doing this, read on... ## A gentle introduction to the Hive connector DevFeed: [A gentle introduction to the Hive connector]() Original publisher: [Read original article]() Author: Brian Olsen Published: 2020-10-20T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Hadoop](), [Amazon S3](), [Cloud](), [data](), [Apache Spark](), [systems](), [parquet](), [CSV](), [JSON]() Tags: [amazon-s3](), [blog](), [cloud](), [cloud-storage](), [code](), [data](), [hadoop](), [json](), [object-storage](), [parquet](), [s3](), [spark](), [systems]() ### AI overview This article gently explains Trino's Hive connector, clarifying that it reads data organized according to Hive conventions without using the Hive runtime. It introduces the main components of Hive architecture and explains how object storage and metadata connect SQL tables to files. ### Source excerpt TL;DR: The Hive connector is what you use in Trino for reading data from object storage that is organized according to the rules laid out by Hive, without using the Hive runtime code. One of the most confusing aspects when starting Trino is the Hive connector. Typically, you seek out the use of Trino when you experience an intensely slow query turnaround from your existing Hadoop, Spark, or Hive infrastructure. In fact, the genesis of Trino, formerly known as Presto, came about due to these slow Hive query conditions at Facebook back in 2012. So when you learn that Trino has a Hive connector, it can be rather confusing since you moved to Trino to circumvent the slowness of your current Hive cluster. Another common source of confusion is when you want to query your data from your cloud object storage, such as AWS S3, MinIO, and Google Cloud Storage. This too uses the Hive connector. If that confuses you, don't worry, you are not alone. This blog aims to explain this commonly confusing nomenclature. ## Hive 3 support in Presto DevFeed: [Hive 3 support in Presto]() Original publisher: [Read original article]() Author: Piotr Findeisen, Starburst Data Published: 2019-12-28T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Hadoop](), [data](), [ci](), [Availability]() Tags: [compatibility](), [continuous-integration](), [hadoop](), [hdfs](), [integration]() ### AI overview This article summarizes Presto's compatibility with Hive 3, including support for Hadoop Erasure Coding, transactional tables, timestamp values stored in ORC, and Hive bucketing v2. It also describes compatibility improvements delivered across Presto releases and ongoing work toward fuller Hive 3 integration. ### Source excerpt The Hive community is centered around a few different Hive distributions, one of them being Hortonworks Data Platform (HDP). Even after the Cloudera-Hortonworks merger there is vivid interest in HDP 3, featuring Hive 3. Presto is ready for the game. In this post, we summarize which Hive 3 features Presto already supports, covering all the work that went into Presto to achieve that. We also outline next steps lying ahead. ## Even Faster ORC DevFeed: [Even Faster ORC]() Original publisher: [Read original article]() Author: Dain Sundstrom, Martin Traverso Published: 2019-04-23T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Optimization](), [Code](), [Hadoop](), [SQL](), [Benchmark](), [Programming]() Tags: [batch](), [benchmark](), [code](), [cpu](), [hadoop](), [optimization](), [performance](), [sql]() ### AI overview This article explains how Trino improved its custom ORC reader. The changes optimize type-specific decoding, bulk reads, null handling, and dynamic dispatch, reducing TPC-DS query time by about 5% and CPU usage by about 9%. ### Source excerpt Trino is known for being the fastest SQL on Hadoop engine, and our custom ORC reader implementation is a big reason for this speed - now it is even faster! ## How to Setup a Scheduled Scala Spark Job DevFeed: [How to Setup a Scheduled Scala Spark Job]() Original publisher: [Read original article]() Published: 2019-03-27T00:00:00Z Content type: tutorial Language: en Sources: [Curalate]() Topics: [Apache Spark](), [Scala](), [Amazon Web Services](), [Amazon S3](), [Hadoop](), [SDKs](), [Command-line interface]() Tags: [aws](), [cli](), [daily](), [data](), [ec2](), [emr](), [github](), [hadoop](), [how-to](), [job](), [pipeline](), [s3](), [scala](), [scheduled](), [sdk](), [spark](), [weekly]() ### AI overview A tutorial for packaging and deploying a Scala Spark job as a fat JAR, uploading it to Amazon S3, and configuring it to run on a schedule through AWS Data Pipeline and EMR. ### Source excerpt Have you written a Scala Spark job that processes a massive amount of data on an intimidating amount of RAM and you want to run it daily/weekly/monthly on a schedule on AWS? I had to do this recently, and couldn't find a good tutorial on the full process to get the spark job running. Included in this article and accompanying repository is everything you need to get your Scala Spark job running on AWS Data Pipeline and EMR. Code Repo This tutorial is not going to walk you through the process of actually writing your specific Scala Spark job to do whatever number crunching you need. There are already plenty of resources available (1, 2, 3) to get you started on that. The code template for setting up a Spark Scala job is available in this GitHub repo. Assuming that you have already written your Spark Job and are only using the AWS Java SDK to connect to your AWS data stores, drop your code in the Main function of SparkJob.scala and run the deploy.sh script to upload the fat jar to your S3 bucket. If you do take other dependencies, then it may take some extra work on your part. To run a Scala Spark job on AWS you need to compile a fat jar that contains the byte code for your job and all of the libraries it needs to run. This project already has the sbt-assembly plugin setup and a assemblyMergeStrategy set up to package the Spark, Hadoop, and AWS SDK together in the fat jar. If you need to add in other libraries that do not play well with each other, or are using a noncompatible version of Spark for this current repo, there are a few good resources available to help you through the needed build.sbt modifications. Outside of the previously mentioned needed changes you need to set a few parameters in the deploy.sh script. Mainly the deploymentPath to your specific S3 bucket, adding a profile to the AWS CLI command to upload to your specific S3 bucket if it's private, and changing the resulting fat jar name if you please. The deploy script uses the AWS CLI to upload the fat ## Announcing OpenTSDB 2.4.0: Rollup and Pre-Aggregation Storage, Histograms, Sketches, and More DevFeed: [Announcing OpenTSDB 2.4.0: Rollup and Pre-Aggregation Storage, Histograms, Sketches, and More]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-12-27T17:01:17Z Content type: release Language: en Sources: [Yahoo]() Topics: [Databases](), [Time Series](), [Hadoop](), [Monitoring](), [Data analysis](), [Open Source](), [Temporal data]() Tags: [data-analysis](), [databases](), [hadoop](), [monitoring](), [open-source](), [release](), [salesforce](), [time-series](), [workflows](), [yahoo-engineering]() ### AI overview OpenTSDB 2.4.0 is released with rollup and pre-aggregation storage for time series data, along with histograms and sketches. The release focuses on retaining lower-resolution data for longer periods, reducing query workloads, and supporting more accurate percentile analysis across multiple series. ### Source excerpt yahoodevelopers: By Chris Larsen, Architect OpenTSDB is one of the first dedicated open source time series databases built on top of Apache HBase and the Hadoop Distributed File System. Today, we are proud to share that version 2.4.0 is now available and has many new features developed in-house and with contributions from the open source community. This release would not have been possible without support from our monitoring team, the Hadoop and HBase developers, as well as contributors from other companies like Salesforce, Alibaba, JD.com, Arista and more. Thank you to everyone who contributed to this release! A few of the exciting new features include: Rollup and Pre-Aggregation Storage As time series data grows, storing the original measurements becomes expensive. Particularly in the case of monitoring workflows, users rarely care about last years' high fidelity data. It's more efficient to store lower resolution "rollups" for longer periods, discarding the original high-resolution data. OpenTSDB now supports storing and querying such data so that the raw data can expire from HBase or Bigtable, and the rollups can stick around longer. Querying for long time ranges will read from the lower resolution data, fetching fewer data points and speeding up queries. Likewise, when a user wants to query tens of thousands of time series grouped by, for example, data centers, the TSD will have to fetch and process a significant amount of data, making queries painfully slow. To improve query speed, pre-aggregated data can be stored and queried to fetch much less data at query time, while still retaining the raw data. We have an Apache Storm pipeline that computes these rollups and pre-aggregates, and we intend to open source that code in 2019. For more details, please visit http://opentsdb.net/docs/build/html/user_guide/rollups.html. Histograms and Sketches When monitoring or performing data analysis, users often like to explore percentiles of their measurements, such as the 9 ## Omid 1.0.0 adds low-latency transaction processing for Apache Phoenix DevFeed: [Omid 1.0.0 adds low-latency transaction processing for Apache Phoenix]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-12-06T19:32:04Z Content type: release Language: en Sources: [Yahoo]() Topics: [phoenix](), [Transactions](), [Hadoop](), [SQL](), [Deployment](), [Cloud]() Tags: [big-data](), [cloud](), [deployment](), [hadoop](), [low-latency](), [open-source](), [phoenix](), [real-time](), [speed](), [sql](), [transactions](), [yahoo-engineering]() ### AI overview This release article introduces Omid 1.0.0, an open source transaction processing platform for Big Data, and announces its selection as the transaction management provider for Apache Phoenix. It describes the Omid Low Latency protocol, which reduces short-transaction latency by 5 times under light load and by 10 to 100 times under heavy load. ### Source excerpt yahoodevelopers: By Ohad Shacham, Yonatan Gottesman, Edward Bortnikov Scalable Systems Research, Verizon/Oath Omid, an open source transaction processing platform for Big Data, was born as a research project at Yahoo (now part of Verizon), and became an Apache Incubator project in 2015. Omid complements Apache HBase, a distributed key-value store in Apache Hadoop suite, with a capability to clip multiple operations into logically indivisible (atomic) units named transactions. This programming model has been extremely popular since the dawn of SQL databases, and has more recently become indispensable in the NoSQL world. For example, it is the centerpiece for dynamic content indexing of search and media products at Verizon, powering a web-scale content management platform since 2015. Today, we are excited to share a new chapter in Omid's history. Thanks to its scalability, reliability, and speed, Omid has been selected as transaction management provider for Apache Phoenix, a real-time converged OLTP and analytics platform for Hadoop. Phoenix provides a standard SQL interface to HBase key-value storage, which is much simpler and in many cases more performant than the native HBase API. With Phoenix, big data and machine learning developers get the best of all worlds: increased productivity coupled with high scalability. Phoenix is designed to scale to 10,000 query processing nodes in one instance and is expected to process hundreds of thousands or even millions of transactions per second (tps). It is widely used in the industry, including by Alibaba, Bloomberg, PubMatic, Salesforce, Sogou and many others. We have just released a new and significantly improved version of Omid (1.0.0), the first major release since its original launch. We have extended the system with multiple functional and performance features to power a modern SQL database technology, ready for deployment on both private and public cloud platforms. A few of the significant innovations include: Protocol ## Hadoop Contributors Meetup at Oath DevFeed: [Hadoop Contributors Meetup at Oath]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-11-08T19:07:25Z Content type: news Language: en Sources: [Yahoo]() Topics: [Hadoop](), [Development](), [Containers](), [Latency](), [Open Source]() Tags: [big-data](), [containers](), [development](), [docker](), [hadoop](), [low-latency](), [open-source](), [yahoo-engineering]() ### AI overview Oath hosted a day-long Hadoop Contributors Meetup at its Sunnyvale campus, bringing together more than 80 Hadoop users, contributors, committers, and PMC members. Talks and breakout sessions covered challenges and solutions across the Hadoop ecosystem, including HDFS, Tez, containers, low-latency processing, deep learning, scalability, security, and cloud deployment. ### Source excerpt yahoodevelopers: By Scott Bush, Director, Hadoop Software Engineering, Oath On Tuesday, September 25, we hosted a special day-long Hadoop Contributors Meetup at our Sunnyvale, California campus. Much of the early Hadoop development work started at Yahoo, now part of Oath, and has continued over the past decade. Our campus was the perfect setting for this meetup, as we continue to make Hadoop a priority. More than 80 Hadoop users, contributors, committers, and PMC members gathered to hear talks on key issues facing the Hadoop user community. Speakers from Ampool, Cloudera, Hortonworks, Microsoft, Oath, and Twitter detailed some of the challenges and solutions pertinent to their parts of the Hadoop ecosystem. The talks were followed by a number of parallel, birds of a feather breakout sessions to discuss HDFS, Tez, containers and low latency processing. The day ended with a reception and consensus that the event went well and should be repeated in the near future. Presentation recordings (YouTube playlist) and slides (links included in the video description) are available here: Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda Tan, Hortonworks Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara, Botong Huang "HDFS Scalability and Security", Daryn Sharp, Senior Engineer, Oath The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, Ampool Moving the Oath Grid to Docker, Eric Badger, Software Developer Engineer, Oath Vespa: Open Source Big Data Serving Engine, Jon Bratseth, Distinguished Architect, Oath Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shane Kumpf, Hortonworks How Twitter Hadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenu Thank you to all the presenters and the attendees both in person and remote! P.S. We're hiring! Learn more about career opportunities at Oath. ## Success at Apache: Migrating Yahoo Jobs from Apache MapReduce to Apache Tez DevFeed: [Success at Apache: Migrating Yahoo Jobs from Apache MapReduce to Apache Tez]() Original publisher: [Read original article]() Author: mikesefanov Published: 2018-02-05T14:43:07Z Content type: article Language: en Sources: [Yahoo]() Topics: [Hadoop](), [Open Source](), [data](), [Apache Spark]() Tags: [2017](), [apache](), [blog](), [bug](), [data](), [distributed](), [hadoop](), [hdfs](), [oath](), [open-source](), [spark](), [tez](), [yahoo](), [yahoo-engineering]() ### AI overview Kuhu Shukla describes her experience contributing to Apache projects at Yahoo and migrating Yahoo jobs from Apache MapReduce to Apache Tez. The article also discusses Oath's Apache-based grid infrastructure and her early work with YARN, HDFS, and Hadoop. ### Source excerpt yahoodevelopers: Kuhu Shukla (bottom center) and team at the 2017 DataWorks Summit By Kuhu Shukla This post first appeared here on the Apache Software Foundation blog as part of ASF's "Success at Apache" monthly blog series. As I sit at my desk on a rather frosty morning with my coffee, looking up new JIRAs from the previous day in the Apache Tez project, I feel rather pleased. The latest community release vote is complete, the bug fixes that we so badly needed are in and the new release that we tested out internally on our many thousand strong cluster is looking good. Today I am looking at a new stack trace from a different Apache project process and it is hard to miss how much of the exceptional code I get to look at every day comes from people all around the globe. A contributor leaves a JIRA comment before he goes on to pick up his kid from soccer practice while someone else wakes up to find that her effort on a bug fix for the past two months has finally come to fruition through a binding +1. Yahoo - which joined AOL, HuffPost, Tumblr, Engadget, and many more brands to form the Verizon subsidiary Oath last year - has been at the frontier of open source adoption and contribution since before I was in high school. So while I have no historical trajectories to share, I do have a story on how I found myself in an epic journey of migrating all of Yahoo jobs from Apache MapReduce to Apache Tez, a then-new DAG based execution engine. Oath grid infrastructure is through and through driven by Apache technologies be it storage through HDFS, resource management through YARN, job execution frameworks with Tez and user interface engines such as Hive, Hue, Pig, Sqoop, Spark, Storm. Our grid solution is specifically tailored to Oath's business-critical data pipeline needs using the polymorphic technologies hosted, developed and maintained by the Apache community. On the third day of my job at Yahoo in 2015, I received a YouTube link on An Introduction to Apache Tez. I watched ## REDstack: An Open-Source Tool for Provisioning Kerberized Hadoop Clusters on OpenStack DevFeed: [REDstack: An Open-Source Tool for Provisioning Kerberized Hadoop Clusters on OpenStack]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2017-12-07T06:00:00Z Content type: article Language: en Sources: [Target]() Topics: [big-data](), [Provisioning](), [Hadoop](), [openstack](), [Orchestration](), [Python](), [Open Source](), [Docker]() Tags: [big-data](), [big-data-infrastructure](), [chef](), [cloud](), [docker](), [druid](), [elasticsearch](), [hadoop](), [open-source](), [openstack](), [orchestration](), [provisioning](), [python]() ### AI overview REDstack is an open-source sandbox tool for Big Data development that provisions kerberized Hadoop clusters on OpenStack. It combines a cookbook for installing and configuring cluster components with a Python orchestration application that manages resource provisioning, Chef deployment, and component installation. ### Source excerpt REDstack is Now Open Source! We are officially open sourcing REDstack, our sandbox tool for Big Data development at Target. What is REDstack? REDstack is a tool for provisioning kerberized clusters on OpenStack. We created it with four goals in mind: Provide a secured environment, with the ability to leverage preconfigured LDAP and Kerberos servers. Out of the box usability, allowing you to log in with preconfigured user accounts. Custom user management utilities to administer the cluster. Provide a fully customizable experience, everything is a configuration option in your build files: Cluster size, node sizes, types of nodes and node roles, Hadoop configurations, heap sizes, and components, All users, passwords, and secure assets. Components REDstack is made up of two major components: hdp-cloud - The cookbook The cookbook is used by the application itself to install components and lay down cluster configuration. The cookbook can be used independently of REDstack to manually provision a cluster. REDstack - The orchestration component REDstack is a python application that performs all of the high-level complexities and timings associated with a full Hadoop installation: Orchestrates the provisioning of resources over OpenStack APIs, Controls and monitors parallel Chef deployment across the cluster, Manages and monitors cluster component install over HTTPS requests. REDstack is bundled with a Docker image, where the configs are set up locally before an installation, and all of the dependencies are updated and configured. How to Get Started Head over to the repository at https://github.com/target/redstack and follow along. The repo has instructions on how to build and configure the clusters using the included Docker image. History of the Project Target's Big Data Platform Team manages multiple Big Data environments, with hundreds of nodes and many PB's of data. As mentioned in our prior blog posts, we depend heavily on Chef as a core part of our CI/CD pipeline. Durin ## From Hadoop and Cassandra to Kafka Streams DevFeed: [From Hadoop and Cassandra to Kafka Streams]() Original publisher: [Read original article]() Author: Nicolas Yann Couturier Published: 2017-07-31T12:00:00Z Content type: article Language: en Sources: [Finn.no]() Topics: [Kafka](), [Apache Cassandra](), [Hadoop]() Tags: [cassandra](), [hadoop](), [kafka]() ### AI overview The article describes FINN.no's existing pipeline for computing classified-ad statistics from user actions published to Apache Kafka, stored in Cassandra, and processed in batches by Hadoop. It introduces the need to join these events with demographic data to provide more detailed viewer statistics. ### Source excerpt Some context People who publish their classified ads on FINN.no get access to various statistics to see how their ads are performing. For a user it can look something like this: Statistics the owner of a realestate ad gets to see The top left bar chart shows the repartition of the incoming traffic by day and the legend to the right of it shows the total numbers for each type of incoming traffic. The lower section is divided in 3 parts: the left part shows the number of views by unique users; the one to the center shows how many users have been notified of the ad by email and how many added the ad to their favorites; the right part shows how many viewers out of the total come from a specific type of traffic. This gives the user basic insight into the reach of their ad, such as how many views it has and how many unique users have viewed it. Around November 2016 there was a request to show more detailed information about the viewers, such as the age, gender and location distribution of the viewers (demographic information) so the owner could potentially change an ad to better fit the audience they wanted. Existing solution As users view ads, do actions (such as send a message to an ad's owner or scroll down and read the whole page, for example), these actions are gathered and published internally on Apache Kafka. These streams of data then become the basis for computing the statistics above. The plan was then to also publish demographic data about the viewers (such as location, age and gender) on Kafka and join the user actions with this demographic data to provide enhanced statistics. At the time, the action events published on Kafka were saved to Cassandra Apache Cassandra clusters, and statistics were being computed as batches on an aging Apache Hadoop cluster reading from Cassandra. Both our Hadoop and Cassandra clusters had not receive much love recently and were all on end-of-life versions. The old system also had an increasing tendency to fail, so we were also i ## Hadoop Rolling Upgrades DevFeed: [Hadoop Rolling Upgrades]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2017-05-22T05:00:00Z Content type: article Language: en Sources: [Target]() Topics: [Hadoop](), [big-data](), [Provisioning](), [Orchestration](), [Cloud](), [Apache Spark]() Tags: [big-data](), [cloud](), [hadoop](), [infrastructure](), [orchestration](), [provisioning](), [spark](), [upgrades]() ### AI overview This article describes Target's move from outage-based Hadoop upgrades to rolling maintenance. It covers the upgrade from Hadoop 2.7.1 to 2.7.3, the goal of avoiding planned downtime, and the use of cluster administration and REDStack provisioning tools to test and orchestrate the process. ### Source excerpt Hadoop upgrades over the last few years meant long outages where the Big Data platform team would shutdown the cluster, perform the upgrade, start services and then complete validation before notifying users it was ok to resume activity. This approach is a typical pattern for major upgrades even outside Target and reduces the complexity and risks associated with the upgrade. While this worked great for the platform team, it was not ideal for the hundreds of users and thousands of jobs that were dependent on the platform. That is why we decided to shake things up and go all in for rolling maintenance. Cluster Details: Hadoop (Core) 2.7.1 to 2.7.3 Mixed cluster workload running Hive, Tez, MR, Spark, Pig, HBase, Storm Goal Our March 2016 Hadoop upgrade was the turning point for rolling maintenance. With a large outage for the upgrade and monthly maintenance windows leading up to it, we decided to challenge ourselves with rolling maintenance to reach our uptime goals for core components. This would allow the platform team to deploy changes faster, reduce maintenance risk by not bundling changes together every month and more importantly not impact users with planned downtime. Drawing the line in the sand for rolling maintenance meant that it was time to get to work on our upgrade strategy. Reviewing the Playbook We started reviewing the upgrade process in November 2016 with the goal to upgrade our first admin cluster in December. The short turnaround time meant that we would leverage our cluster administration tool to handle the upgrade orchestration. The focus shifted towards understanding the order of events and evaluating the process for potential impacts. Running through the upgrades required a quick way to iterate through Hadoop cluster deployments and then tearing them down to retest. This is where REDStack, which is our internal cloud provisioning tool to build out a secure Hadoop cluster with production like configurations, came into play. After going through the ## How Target Performance Tunes Machine Learning Applications DevFeed: [How Target Performance Tunes Machine Learning Applications]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2016-09-29T05:00:00Z Content type: article Language: en Sources: [Target]() Topics: [Machine learning](), [distributed-systems](), [Apache Spark](), [Hadoop](), [Scalability](), [Filesystems](), [datasets](), [Algorithms](), [Transactions](), [Open Source]() Tags: [algorithms](), [apache-spark](), [data](), [data-science-and-engineering](), [distributed](), [engineering](), [hadoop](), [machine-learning](), [open-source](), [scalability](), [science]() ### AI overview The article discusses improving the performance and relevance of machine-learning applications by evaluating matrix multiplication at scale. It describes distributed computing requirements, distributed file storage, scalability, fault tolerance, and the use of Apache Hadoop, Apache Spark, and Apache Mahout. ### Source excerpt At Target we aim to make shopping more fun and relevant for our guests through extensive use of data - and believe me, we have lots of data! Tens of millions of guests and hundreds of thousands of items lead to billions of transactions and interactions. We regularly employ a number of different machine learning techniques on such large datasets for dozens of algorithms. We are constantly looking for ways to improve speed and relevance of our algorithms and one such quest brought us to carefully evaluate matrix multiplications at scale - since that forms the bedrock for most algorithms. If we make matrix multiplication more efficient, we can speed up most of our algorithms! Before we dig in, let me describe some properties of the landscape we will be working in. First, what do I mean by large scale? A large scale application, at a minimum, will require its computation to be spread over multiple nodes of a distributed computing environment to finish in a reasonable amount of time. These calculations will use existing data that are stored on a distributed file system that provides high-throughput access from the computing environment. Scalability, in terms of storage and compute, should grow as we add to these resources. As the system grows larger and more complex, failures will become more commonplace. Thus, software should be fault-tolerant. Fortunately, there is a lot of existing open-source software that we can leverage to work in such an environment, particularly Apache Hadoop for storing and interacting with our data, Apache Spark as the compute engine, and both Apache Spark and Apache Mahout for applying and building distributed machine learning algorithms. There are many other tools that we can add to the mix as well, but for the purposes of this post we will limit our discussion to these three. With that out of the way, lets dig in! Don't Forget the Basics Begin with good old paper and pencil. Yeah, I know this is about large scale matrix operations that you c ## Bare Metal Big Data Builds DevFeed: [Bare Metal Big Data Builds]() Original publisher: [Read original article]() Author: Target Brands, Inc Published: 2016-01-13T06:00:00Z Content type: article Language: en Sources: [Target]() Topics: [Hadoop](), [Automation](), [openstack](), [Nova](), [Jenkins](), [Bash](), [Python](), [API]() Tags: [ambari](), [api](), [automation](), [bare-metal](), [bash](), [big-data](), [chef](), [hadoop](), [infrastructure](), [ironic](), [jenkins](), [nova](), [openstack](), [python]() ### AI overview This engineering article describes Target's transition from manually building on-premise Hadoop clusters to an automated bare-metal provisioning workflow. It combines OpenStack Ironic and the Nova client with user-data bash scripts and Chef cookbooks to bootstrap servers, configure node roles, and add them to clusters. ### Source excerpt When you first think about scaling an on-premise Hadoop cluster your mind jumps to the process and the teams involved in building the servers, the time needed for configuring them and then the stability required while getting them into the cluster. Here at Target that process used to be measured in months. The story below outlines our journey around scaling our Hadoop cluster, taking the months to hours and adding hundreds of servers in a couple weeks. The Need Early 2013 taught us the lesson that manually managing Hadoop clusters, no matter how small is a time consuming and very repetitive task. Our next cluster build in 2014 drove the adoption of Chef, Artifactory and Jenkins to help with cluster operations. We stood up those components and created new role cookbooks to manage everything on the OS (configurations, storage, Kerberos, MySQL, etc.). While this was a step in the right direction, it left us with a manual process to still create the initial base server build and then add it to the cluster after configuring it with Chef. Build Foundation Closing the gap in our automation meant finding a way to deliver on true end to end builds, from an initial bootstrap to running jobs in your cluster. OpenStack's Ironic project was the first piece of the puzzle. Ironic gives us the ability to provision bare metal servers, similar to how OpenStack automated the VM build process. With Ironic as the foundation, we leveraged the Nova client to manage our instance builds. The nova python client interacts with the Compute service's API, giving us an easy way to specify our build parameters and spinning up an instance on one of our physical servers. The other key piece with the nova client is the ability to send boot information to the server using user data. The bash script sent executes several commands to install the Chef client, setup public keys and run the initial knife bootstrap to set the run list for the build. Example nova boot command: nova boot --image $image_name ## Running Apache Flink on Amazon Elastic Mapreduce DevFeed: [Running Apache Flink on Amazon Elastic Mapreduce]() Original publisher: [Read original article]() Published: 2016-01-06T08:15:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [apache-flink](), [Amazon Web Services](), [Hadoop](), [Scalding](), [data-processing](), [Amazon S3](), [Apache Spark]() Tags: [apache-flink](), [aws](), [data-processing](), [ec2](), [emr](), [flink](), [hadoop](), [hdfs](), [linux](), [realtime](), [s3](), [scala](), [scalding](), [sclading](), [spark](), [streaming]() ### AI overview A tutorial on running Apache Flink jobs on Amazon EMR, including adapting Scalding jobs and launching and accessing an EMR cluster. It also discusses EMRFS support for reading data from Amazon S3. ### Source excerpt I love really Amazon EMR. Over the years it's grown from being "Hadoop on-demand" to a full-fledged cluster management system for running OSS big-data apps (Hadoop MR of course, but also Spark, Hue, Hive, Pig, Oozie and more). While Hadoop out of the box supports reading from S3, EMR has a proprietary implementation called EMRFS that has some nice features. For those reasons, it's really the best Hadoop cluster to use if you're storing your data in S3. Lately I've been experimenting a lot with Apache Flink to replace MR as the excution fabric. At work, we have many, many jobs written in Scalding. Flink can execute Scalding jobs with some very simple modifications which was a great way to move our jobs from MR to a more memory-centric data processing model. However, we really wanted to run our jobs on EMR using Flink. Flink is not an option for EMR (yet) but can we still get our jobs to run? Let's see! Start an EMR cluster The first thing we need is an EMR cluster. You can launch a small test cluster for very cheap. Once it's running, let's ssh onto it and see what's going on. ihummel at mm-mac-3270 in ~ $ ssh hadoop@ec2-54-226-25-85.compute-1.amazonaws.com Last login: Wed Jan 6 16:33:01 2016 from 172.85.47.138 __| __|_ ) _| ( / Amazon Linux AMI ___|\___|___| https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/ 23 package(s) needed for security, out of 49 available Run "sudo yum update" to apply all updates. EEEEEEEEEEEEEEEEEEEE MMMMMMMM MMMMMMMM RRRRRRRRRRRRRRR E::::::::::::::::::E M:::::::M M:::::::M R::::::::::::::R EE:::::EEEEEEEEE:::E M::::::::M M::::::::M R:::::RRRRRR:::::R E::::E EEEEE M:::::::::M M:::::::::M RR::::R R::::R E::::E M::::::M:::M M:::M::::::M R:::R R::::R E:::::EEEEEEEEEE M:::::M M:::M M:::M M:::::M R:::RRRRRR:::::R E::::::::::::::E M:::::M M:::M:::M M:::::M R:::::::::::RR E:::::EEEEEEEEEE M:::::M M:::::M M:::::M R:::RRRRRR::::R E::::E M:::::M M:::M M:::::M R:::R R::::R E::::E EEEEE M:::::M MMM M:::::M R:::R R::::R EE:::::EEEEEEEE::::E ## Getting started with Scalding DevFeed: [Getting started with Scalding]() Original publisher: [Read original article]() Published: 2015-12-20T08:15:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [Scalding](), [data-processing](), [Hadoop](), [Scala](), [Testing](), [Compression](), [Logging]() Tags: [beginners](), [build](), [building](), [compression](), [getting-started](), [hadoop](), [hdfs](), [logging](), [scala](), [scalding](), [sclading](), [testing](), [unit-testing]() ### AI overview A practical guide to starting Scalding projects with SBT. It presents a minimal project structure and dependencies, explains local Hadoop execution and logging, and shows how to test jobs locally before submitting them to a cluster. ### Source excerpt I've been using Scalding for the last few years and really love how simple it makes writing scalalbe data processing jobs. I think many of the issues beginners have with Scalding relate to project setup. I hope this post simplifies things for people so they can started with less hassle. Building your project with SBT The official getting started guide is pretty terse. For our purposes, all you need to get started is a very simple SBT project. simple-scalding-example/ |-- build.sbt |-- project | |-- build.properties | `-- plugins.sbt |-- src | `-- main | `-- scala | `-- WordCountJob.scala `-- log4j.properties The most important thing the setup of the build.sbt file. Our example has minimal dependencies: libraryDependencies ++= Seq( "com.twitter" %% "scalding-core" % "0.15.0", "org.apache.hadoop" % "hadoop-client" % "2.2.0" % "provided", "org.slf4j" % "slf4j-log4j12" % "1.7.13" % "provided" ) The hadoop-client jar is included for compilation, unit testing and running locally on your laptop but will not be pacakged into the final "fat jar" assembly shipped out to the cluster. Additionally, a logging framework is included so that we get good logs when debugging locally. A simple log4j.properties file is included in the repo. The whole job can be run locally using real Hadoop libraries without any additional software install: $ sbt 'run net.themodernlife.WordCount --hdfs --input build.sbt --output target/output' It's even possible to get things like LZO-compression to work with this setup. The main win here is that there is a very simple, repeatable development environment that can be used to test all aspects of the job locally before sending things off to the cluster. ## Running Scalding jobs on Apache Flink DevFeed: [Running Scalding jobs on Apache Flink]() Original publisher: [Read original article]() Published: 2015-12-20T08:15:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [Scalding](), [apache-flink](), [Scala](), [Streaming](), [Hadoop]() Tags: [apache-flink](), [flink](), [hadoop](), [hdfs](), [realtime](), [scala](), [scalding](), [sclading](), [streaming]() ### AI overview This tutorial explains how to run Scalding jobs on Apache Flink. It outlines the required compatibility work, including Cascading 3 support, a Scalding version compiled against Cascading 3, selection of hadoop2-flink as the Cascading fabric, and a fix for a Twitter Chill bug. It also provides a sample Scala word-count job and build configuration changes. ### Source excerpt My previous post showed a very simple Scalding workflow. Apache Flink is a real time streaming framework that's very promising. It also supports running Cascading workflows with very little modification. Surely there must be some way to run a Scalding job on top of Flink? Turns out... YES! In a nutshell Here are the high-level things we need to solve for We need a Scalding job to test this out with cascading-flink requires Cascading 3 We need a new version of Scalding - Compiled against Cascading 3 - Allows hadoop2-flink to be selected as the Cascading "fabric" There is a bug in Twitter Chill We need Flink packaged with the Chill fix Scalding job Let's start with a very simple Scalding job. You can download https://github.com/themodernlife/simple-scalding-example for some inspiration. package net.themodernlife import com.twitter.scalding._ class WordCount(args: Args) extends Job(args) { def tokenize(text: String): Array[String] = { text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("\\s+") } val input = args("input") val output = args("output") TextLine(args("input")) .flatMap[String, String]('line -> 'word)(tokenize) .groupBy('word)(_.size) .write(Tsv(output)) } We'll be making some updates to our build.sbt. Here's what we're starting with organization := "net.themodernlife" name := "simple-scalding-example" scalaVersion := "2.11.7" scalacOptions ++= Seq("-encoding", "utf-8", "-deprecation", "-unchecked", "-feature") resolvers ++= Seq( "Concurrent Maven Repo" at "http://conjars.org/repo", "Twitter Maven Repo" at "http://maven.twttr.com" ) libraryDependencies ++= Seq( "com.twitter" %% "scalding-core" % "0.15.0", "org.apache.hadoop" % "hadoop-client" % "2.2.0" % "provided", "org.slf4j" % "slf4j-log4j12" % "1.7.13" % "provided" ) A new Scalding build We need to update Scalding Apply https://github.com/twitter/scalding/pull/1446 Apply/hack https://github.com/twitter/scalding/pull/1220 Hack Build.scala to ignore some submodules According to http://www.cascading.org/2 [Next page]()