# Hadoop Published articles for Hadoop. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering DevFeed: [The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-08T12:16:09Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [pandas](), [data-engineering](), [Artificial Intelligence](), [DuckDB](), [data-platforms](), [software-development](), [future of software](), [Apache Spark](), [Hadoop](), [Open Source](), [parquet]() Tags: [ai](), [apache-arrow](), [arrow](), [data](), [data-engineering](), [data-platforms](), [duckdb](), [hadoop](), [large-language-models](), [open-source](), [pandas](), [parquet](), [software-development]() ### AI overview An interview with Wes McKinney covers the origins of pandas and Apache Arrow, the evolution of modern data engineering from Hadoop to lakehouse architectures, and the roles of tools such as Parquet, DuckDB, DataFusion, and Spark. McKinney also discusses how AI affects software development, arguing that it can improve experienced engineers' productivity but does not replace software engineering, architecture, or judgment. ### Source excerpt interview with Wes McKinney ## Beyond the warehouse: How METRO Markets built a do-it-all data platform on ClickHouse Cloud DevFeed: [Beyond the warehouse: How METRO Markets built a do-it-all data platform on ClickHouse Cloud]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-06-18T20:31:40Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Cloud](), [data](), [data-engineering](), [observability](), [real-time](), [Hadoop](), [log management](), [BigQuery](), [Artificial Intelligence]() Tags: [analytics](), [bigquery](), [clickhouse](), [cloud](), [data](), [data-engineering](), [hadoop](), [infrastructure](), [logging](), [observability](), [real-time]() ### AI overview METRO Markets replaced its self-hosted Hadoop-based data stack with ClickHouse Cloud, consolidating company data into a single warehouse that supports data warehousing, real-time seller analytics, credit risk modeling, observability, operational logging, and new AI use cases. ### Source excerpt How METRO Markets replaced a failing Hadoop-based stack with ClickHouse Cloud to build a single platform now powering data warehousing, real-time seller analytics, credit risk modeling, observability, and AI across the whole company. ## Why HTAP Databases Are Giving Way to Disaggregated Architectures DevFeed: [Why HTAP Databases Are Giving Way to Disaggregated Architectures]() Original publisher: [Read original article]() Author: Zhou Sun Published: 2025-05-04T10:00:00Z Content type: article Language: en Sources: [Blog -- Neon Docs]() Topics: [Databases](), [olap](), [Cloud](), [Latency](), [consistency](), [NoSQL](), [MongoDB](), [Hadoop](), [hdfs](), [CockroachDB](), [Amazon Redshift](), [vitess]() Tags: [cloud](), [cloud-data](), [cockroachdb](), [consistency](), [data-lake](), [databases](), [engineering](), [hadoop](), [hdfs](), [latency](), [mongodb](), [olap](), [redshift](), [sql](), [vitess]() ### AI overview The article traces the separation of transactional and analytical database workloads, explaining how differing storage and scaling requirements led to specialized OLTP and OLAP systems. It argues that HTAP as a single database architecture is declining while its underlying ideas persist in today's disaggregated data stack. ### Source excerpt This blog is inspired by Jordan Tigani's "Big Data is Dead." Jordan and I actually spent some time building an HTAP database at SingleStore. From the one database that did everything in the '80s, to the great divide, to HTAP, to today's disaggregated stack--here's why HTAP as a database is dead, but its spirit lives on. ## Out with the old file system DevFeed: [Out with the old file system]() Original publisher: [Read original article]() Author: Manfred Moser, David Phillips, Mateusz Gajewski Published: 2025-02-10T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Filesystems](), [Apache Iceberg](), [Hadoop](), [Amazon S3](), [Azure](), [Google Cloud Platform (GCP)]() Tags: [amazon-s3](), [apache-iceberg](), [apache-parquet](), [azure](), [compression](), [data-lake](), [deprecated](), [google-cloud](), [hadoop](), [maintenance](), [release]() ### AI overview Trino 470 deprecated its legacy Hadoop-based file system support, which will be removed in a future release. The article explains Trino's move toward custom file system implementations for cloud storage and describes the migration path for users, including catalog-level file system configuration and warnings for deprecated properties. ### Source excerpt What a long journey it has been! From the start Trino supported querying Hive data and used libraries from the Hive and Hadoop ecosystem. With the release of Trino 470 we mark another milestone to more features and better performance for data lake and lakehouse querying with Trino. We deprecated the legacy file system support, and will permanently remove them in an upcoming release. ## The long journey to Apache Ranger DevFeed: [The long journey to Apache Ranger]() Original publisher: [Read original article]() Author: Manfred Moser Published: 2024-12-02T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [releases]() Tags: [analytics](), [apache](), [data](), [hadoop](), [integration](), [pull-request](), [release](), [testing]() ### AI overview Trino 466 adds an Apache Ranger plugin for data access control. The article recounts the history of the integration, including prior pull requests, container-image support for testing, and renewed work in 2024. ### Source excerpt Apache Ranger has arrived! With the new Trino 466 you all get another jam-packed release of Trino awesomeness. One of the goodies is a new plugin for access control for your data with Apache Ranger, and it has gone through a long story to get here. Apache Ranger has a long history and wide adoption as an access control system for data lakes using Hadoop and Hive. Since Trino brings fast analytics to this space, and also supports modern data lakehouses and other data sources, Apache Ranger is a natural fit for access control on a Trino-powered data platform. ## How Uber Reduced Their Log Size By 99% DevFeed: [How Uber Reduced Their Log Size By 99%]() Original publisher: [Read original article]() Author: Richard Oliver Bray Published: 2024-10-09T13:02:55Z Content type: article Language: en Sources: [Hacking Scale by Better Stack]() Topics: [Logging](), [Hadoop](), [Filesystems](), [big-data](), [Apache Spark](), [data]() Tags: [apache-spark](), [big-data](), [cli](), [clickhouse](), [data](), [hadoop](), [hdfs](), [logging]() ### AI overview The article explains how Uber addressed the storage cost of generating roughly 5 PB of INFO-level logs each month. It describes Uber's use of HDFS and related data-processing tools, while reporting that the company reduced log storage size by 99%. ### Source excerpt Uber broke apart an open source tool to massively compress their logs ## The crushing success of relational databases DevFeed: [The crushing success of relational databases]() Original publisher: [Read original article]() Author: Andy Hattemer Published: 2024-08-13T20:27:05Z Content type: opinion Language: en Sources: [Blog -- Neon Docs]() Topics: [Databases](), [SQL](), [Hadoop](), [Graphs]() Tags: [databases](), [graphs](), [hadoop](), [postgres](), [relational-databases](), [sql]() ### AI overview The article argues that relational databases remain dominant because they repeatedly absorb capabilities associated with competing data models and query systems. It presents document, vector, and graph functionality as features that can coexist within a relational database, and revisits arguments that the relational model will continue to outlast attempted replacements. ### Source excerpt Every once in a while, a revolutionary product comes along and changes everything. And today, we're talking about three of these phenomenal products. The first one is a document store. The second is a vector database. And the third is a graph database. So, three things: document... ## Expanded Memory and Compute with Heroku's New Larger Dynos DevFeed: [Expanded Memory and Compute with Heroku's New Larger Dynos]() Original publisher: [Read original article]() Author: Ethan Limchayseng Published: 2024-03-28T02:25:00Z Content type: release Language: en Sources: [Heroku]() Topics: [Heroku](), [cpu](), [Cache](), [big-data](), [Data analysis]() Tags: [big-data](), [cache](), [cli](), [cloud-infrastructure](), [compute](), [cpu](), [data](), [distributed](), [dynos](), [hadoop](), [heroku](), [how-to](), [machine-learning](), [memory](), [news](), [performance-optimization](), [pricing](), [private-spaces](), [real-time](), [spark](), [video]() ### AI overview Heroku introduces nine larger dyno types across its Performance, Private, and Shield tiers, adding higher memory and CPU limits for compute-intensive workloads. The new sizes support use cases including real-time analytics, caching, machine learning, video encoding, and simulations. ### Source excerpt Introduction Heroku is excited to introduce nine new dyno types to our fleets and product offerings. In 2014, we introduced Performance-tier dynos, giving our customers fully dedicated resources to run their most compute-intensive workloads. Now in 2024, today's standards are rapidly increasing as complex applications and growing data volumes consume more memory and carry heavier [...] The post Expanded Memory and Compute with Heroku's New Larger Dynos appeared first on Heroku. ## Implementing Data Validation with Great Expectations in Hybrid Environments DevFeed: [Implementing Data Validation with Great Expectations in Hybrid Environments]() Original publisher: [Read original article]() Author: Kamila Widyanto Full time DevOps; Site Reliability Engineer; Part Time Rubberduck Linkedin Profile Published: 2023-04-25T00:00:00Z Content type: tutorial Language: en Sources: [Trivago]() Topics: [data-processing](), [Hadoop](), [hdfs](), [integrity](), [Apache Spark](), [configuration](), [Library](), [Python](), [JSON](), [Open Source](), [YAML](), [version-control]() Tags: [configuration](), [data](), [data-pipeline](), [data-processing](), [data-validation](), [devops](), [distributed](), [hadoop](), [hdfs](), [integrity](), [json](), [library](), [open-source](), [python](), [version-control](), [workflow](), [yaml]() ### AI overview This article describes implementing Great Expectations for data validation in a hybrid Hadoop environment. It explains the framework's core concepts and how the authors ran it as a PySpark job in an automated data pipeline, including configuring the Data Context for HDFS constraints. ### Source excerpt Data validation is an essential step in any data processing pipeline, as it ensures the integrity and accuracy of the data to be used across all subsequent processing steps. ## Journey to Iceberg with Trino DevFeed: [Journey to Iceberg with Trino]() Original publisher: [Read original article]() Author: JaeChang Song, Jennifer Oh, Brian Olsen Published: 2022-12-19T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Apache Iceberg](), [data-engineering](), [big-data](), [dashboards](), [Hadoop]() Tags: [data-engineering](), [engineering](), [hadoop](), [hdfs](), [logs](), [metrics](), [open-source](), [performance](), [post](), [scale](), [speed](), [summit](), [switching]() ### AI overview SK Telecom describes its journey from Hive-based Trino deployments to Iceberg after encountering scaling and performance problems. The company used Trino across Hadoop and HDFS-based data platforms, collected query plans, JMX statistics, system metrics, and logs, and built a dashboard to investigate blocked queries and cluster behavior. ### Source excerpt This post comes from the second half of Trino Summit 2022 session. Our friends JaeChang and Jennifer from SK Telecom traveled across the globe from South Korea to join us in person! SK Telecom recently had some issues scaling Trino on the Hive model, among other issues that come with Hive. While some initial tweaking helped speed things up, it ultimately never solved the problem. After switching to Iceberg, SK Telecom ran initial performance tests with some very impressive results. In this talk, Jennifer and JaeChang describe their journey to Iceberg with Trino. ## Building A Modern Data Stack for QazAI DevFeed: [Building A Modern Data Stack for QazAI]() Original publisher: [Read original article]() Author: Baurzhan Kuspayev Published: 2022-06-08T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [data](), [Amazon S3]() Tags: [analytics](), [architecture](), [building](), [clickhouse](), [data](), [databases](), [hadoop](), [monitoring](), [open-source](), [s3](), [speed](), [sql]() ### AI overview QazAI describes moving from a data stack built around S3, Hive, and Clickhouse toward Trino for faster, lower-cost analytics and ETL exploration. The article highlights Trino's SQL support, federated queries, setup simplicity, and a lack of fault tolerance for ETL pipelines. ### Source excerpt At QazAI, we build data lakes as a service for companies. In the original architecture, we get raw data in S3, transform the S3 data with Hive, and then delivered the data to business units via our datamart built on Clickhouse (for optimal delivery speeds). Over time, we were dragged down by the slower speeds and high costs of running Hive, and started shopping for a faster and cheaper open source engine to do our ETL data transformations. ## A Serverless Data Engineering Stack at Teamwork Using Go and AWS DevFeed: [A Serverless Data Engineering Stack at Teamwork Using Go and AWS]() Original publisher: [Read original article]() Author: Joe Minichino Published: 2021-10-19T13:58:52Z Content type: article Language: en Sources: [Teamwork]() Topics: [data-engineering](), [Serverless](), [Go Language](), [Amazon Web Services](), [AWS Lambda](), [Amazon S3](), [GitHub Actions](), [Docker](), [Apache Spark](), [airflow]() Tags: [airflow](), [aws](), [aws-athena](), [aws-lambda](), [data](), [data-engineering](), [docker](), [github-actions](), [go](), [golang](), [hadoop](), [s3](), [serverless](), [serverless-architecture]() ### AI overview This article describes Teamwork's serverless-oriented data engineering stack and the reasons for choosing it. It discusses AWS services including S3, Athena, Glue, QuickSight, Kinesis, and Lambda, with Go used for Lambda processing and GitHub Actions used for deployment. ### Source excerpt A real data lake. Traditional Data Engineering relies on products such as Airflow, Hadoop, Spark and Spark-based architectures, or similar technologies. These are still viable solutions for a number of reason, not least the fact that Data Engineers are few and far between, and the vast majority of them will be familiar in the above technologies or similar products/frameworks. Go Serverless I wrote an article about our tech stack, which includes S3, Athena, Glue, Quicksight, Kinesis etc. What is not immediately apparent is the serverless nature of our stack, which was a deliberate choice taken in the context of a Data Analytics department which was started as an experiment and had to pick its battles very wisely. Sysops, cluster / server management, CI/CD were not top of our list. Creating dashboards was. Also before Teamwork I had a nearly 2-year run working with a company that was entirely serverless in their set up AND mentality, and it was a career-changing experience. We worked prevalently with AWS Lambda when Lambda was a new toy, and we loved it. Finally, despite the "buzz" over Functional Programming being seemingly over, I am an arduous fanatic of it and of what it represents philosophically, a way to represent each problem in terms of an input, some transformation, an output and when possible, no side effects. NOTE: some of the tools I mention are not strictly serverless as much as they are fully managed (eg. Kinesis or Github Actions), but they still involve little or no sysops / devops. Serverless Processing: AWS Lambda Once upon a time you would have to settle for Node.js or Python to write Lambda code, but nowadays not only you can use a whole lot of runtimes, you can simply use your own docker image et voila', you're ready to go. In our case we are ready to Go, since at Teamwork Go(lang) is our programming language of choice. It's fast, easy to use, very safe and reliable, and given that it can complete the same operation faster than most other program ## Плагин Big Data Tools теперь поддерживает IntelliJ IDEA Ultimate, PyCharm Professional, DataGrip 2021.3 EAP и DataSpell DevFeed: [Плагин Big Data Tools теперь поддерживает IntelliJ IDEA Ultimate, PyCharm Professional, DataGrip 2021.3 EAP и DataSpell]() Original publisher: [Read original article]() Author: olegchir (JetBrains) Published: 2021-09-28T06:17:21Z Content type: release Language: ru Sources: [JetBrains RU]() Topics: [Apache Spark](), [ide](), [IntelliJ IDEA](), [PyCharm](), [Data Science](), [Amazon S3](), [Kafka](), [Python]() Tags: [amazon](), [apache-kafka](), [apache-spark](), [apache-zeppelin](), [aws](), [aws-s3](), [configuration](), [hadoop](), [ide](), [intellij-idea](), [kafka](), [pycharm](), [python](), [s3](), [spark](), [zeppelin]() ### AI overview JetBrains released a new Big Data Tools plugin build compatible with IntelliJ IDEA Ultimate and PyCharm Professional 2021.3, with planned support for DataGrip 2021.3 EAP and support for running in DataSpell. The update adds features including Spark Submit run configurations, Kafka monitoring, AWS S3 named profiles, Zeppelin notebook search, and Python interpreter selection. ### Source excerpt Недавно мы выпустили новую сборку плагина Big Data Tools, совместимую со свежими (2021.3) версиями IntelliJ IDEA Ultimate и PyCharm Professional. Когда в октябре выйдет DataGrip 2021.3, эта сборка тоже будет с ним работать. Более того, теперь мы умеем запускаться в DataSpell -- новой IDE для Data Science. Если вы используете старые версии Big Data Tools, сейчас самое время обновиться и попробовать новую версию плагина вместе со свежей версией IDE! В этом году мы много чего улучшили и добавили совершенно новые фичи (например, запуск Spark Submit в виде Run Configuration). Вот небольшой список изменений за этот год. Этот список -- лишь небольшая капля в море того, что изменилось с прошлого года. Читать далее ## Обзор плагина Big Data Tools DevFeed: [Обзор плагина Big Data Tools]() Original publisher: [Read original article]() Author: olegchir (JetBrains) Published: 2021-07-28T10:41:36Z Content type: article Language: ru Sources: [JetBrains RU]() Topics: [ide](), [big-data](), [Hadoop](), [Apache Spark](), [data](), [Scala](), [Python]() Tags: [big-data](), [big-data-tools](), [data](), [hadoop](), [ide](), [jetbrains](), [kafka](), [python](), [scala](), [spark](), [tools](), [zeppelin]() ### AI overview This article reviews JetBrains' Big Data Tools plugin for working with cloud file systems, Hadoop, Spark, and Zeppelin directly from an IDE. It explains the plugin's role in data-engineering workflows, including ETL, and notes support for Scala and Python. ### Source excerpt Храните файлы в облачных файловых системах или, может быть, используете Hadoop, Spark и Zeppelin? А пробовали ли вы работать с ними напрямую из IDE? Привет, меня зовут Олег, я из команды плагина Big Data Tools. В этой статье мы поговорим, зачем этот плагин нужен, как применяется и где его достать. За последний год плагин прошёл большой путь и из экспериментального продукта превратился в боевое решение, на которое стоит взглянуть специалистам по Big Data. В JetBrains мы создаем IDE и другие инструменты, которые делают жизнь разработчиков лучше. Big Data Tools -- это очень узкоспециализированный, редкоземельный плагин, который предназначен для конкретного вида разработчиков -- для дата-инженеров. Если вам интересно подробней узнать о мире Big Data и работе дата-инженеров, рекомендую развернутую серию статей Паши Финкельштейна. Здесь мы рассмотрим одну из самых популярных схем. Читать далее ## Trino on ice I: A gentle introduction To Iceberg DevFeed: [Trino on ice I: A gentle introduction To Iceberg]() Original publisher: [Read original article]() Author: Brian Olsen Published: 2021-05-03T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Apache Iceberg]() Tags: [architecture](), [big-data](), [hadoop](), [open-source](), [spark]() ### AI overview An introduction to Apache Iceberg's table format with Trino, framed by the limitations and informal conventions of the Hive connector model. ### Source excerpt Welcome to the Trino on ice series, covering the details around how the Iceberg table format works with the Trino query engine. The examples build on each previous post, so it's recommended to read the posts sequentially and reference them as needed later. Here are links to the posts in this series: Trino on ice I: A gentle introduction to Iceberg Trino on ice II: In-place table evolution and cloud compatibility with Iceberg Trino on ice III: Iceberg concurrency model, snapshots, and the Iceberg spec Trino on ice IV: Deep dive into Iceberg internals Back in the Gentle introduction to the Hive connector blog post, I discussed a commonly misunderstood architecture and uses of the Trino Hive connector. In short, while some may think the name indicates Trino makes a call to a running Hive instance, the Hive connector does not use the Hive runtime to answer queries. Instead, the connector is named Hive connector because it relies on Hive conventions and implementation details from the Hadoop ecosystem - the invisible Hive specification. ## We're rebranding PrestoSQL as Trino DevFeed: [We're rebranding PrestoSQL as Trino]() Original publisher: [Read original article]() Author: Martin Traverso, Dain Sundstrom, David Phillips Published: 2020-12-27T00:00:00Z Content type: release Language: en Sources: [Trino Blog]() Topics: [Software](), [Open Source](), [Data Infrastructure](), [Hadoop](), [GitHub](), [Slack]() Tags: [community](), [contributors](), [data-infrastructure](), [github](), [hadoop](), [infrastructure](), [open-source](), [rebranding]() ### AI overview The article announces that PrestoSQL is being rebranded as Trino. It explains that the software and community remain intact while describing the project's origins in low-latency analytics over Hadoop data and its commitment to an open, independent, collaborative community. ### Source excerpt We're rebranding PrestoSQL as Trino. The software and the community you have come to love and depend on aren't going anywhere, we are simply renaming. Trino is the new name for PrestoSQL, the project supported by the founders and creators of Presto® along with the major contributors - just under a shiny new name. And now you can find us here: GitHub: https://github.com/trinodb/trino. Please give it a star! Twitter: @trinodb Slack: https://trino.io/slack.html If you want to learn why we're doing this, read on... ## A gentle introduction to the Hive connector DevFeed: [A gentle introduction to the Hive connector]() Original publisher: [Read original article]() Author: Brian Olsen Published: 2020-10-20T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Hadoop](), [Amazon S3](), [Cloud](), [data](), [Apache Spark](), [systems](), [parquet](), [CSV](), [JSON]() Tags: [amazon-s3](), [blog](), [cloud](), [cloud-storage](), [code](), [data](), [hadoop](), [json](), [object-storage](), [parquet](), [s3](), [spark](), [systems]() ### AI overview This article gently explains Trino's Hive connector, clarifying that it reads data organized according to Hive conventions without using the Hive runtime. It introduces the main components of Hive architecture and explains how object storage and metadata connect SQL tables to files. ### Source excerpt TL;DR: The Hive connector is what you use in Trino for reading data from object storage that is organized according to the rules laid out by Hive, without using the Hive runtime code. One of the most confusing aspects when starting Trino is the Hive connector. Typically, you seek out the use of Trino when you experience an intensely slow query turnaround from your existing Hadoop, Spark, or Hive infrastructure. In fact, the genesis of Trino, formerly known as Presto, came about due to these slow Hive query conditions at Facebook back in 2012. So when you learn that Trino has a Hive connector, it can be rather confusing since you moved to Trino to circumvent the slowness of your current Hive cluster. Another common source of confusion is when you want to query your data from your cloud object storage, such as AWS S3, MinIO, and Google Cloud Storage. This too uses the Hive connector. If that confuses you, don't worry, you are not alone. This blog aims to explain this commonly confusing nomenclature. ## Hive 3 support in Presto DevFeed: [Hive 3 support in Presto]() Original publisher: [Read original article]() Author: Piotr Findeisen, Starburst Data Published: 2019-12-28T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Hadoop](), [data](), [ci](), [Availability]() Tags: [compatibility](), [continuous-integration](), [hadoop](), [hdfs](), [integration]() ### AI overview This article summarizes Presto's compatibility with Hive 3, including support for Hadoop Erasure Coding, transactional tables, timestamp values stored in ORC, and Hive bucketing v2. It also describes compatibility improvements delivered across Presto releases and ongoing work toward fuller Hive 3 integration. ### Source excerpt The Hive community is centered around a few different Hive distributions, one of them being Hortonworks Data Platform (HDP). Even after the Cloudera-Hortonworks merger there is vivid interest in HDP 3, featuring Hive 3. Presto is ready for the game. In this post, we summarize which Hive 3 features Presto already supports, covering all the work that went into Presto to achieve that. We also outline next steps lying ahead. ## A Case Study on Anaconda: How it Delivers its Platform DevFeed: [A Case Study on Anaconda: How it Delivers its Platform]() Original publisher: [Read original article]() Author: info@goteleport.com (Jon Silvers) Published: 2019-08-29T00:00:00Z Content type: article Language: en Sources: [Teleport]() Topics: [Kubernetes](), [Data Science](), [Deployment](), [Cloud](), [Hadoop](), [Apache Spark]() Tags: [case-study](), [cloud-native](), [data-science](), [deployment](), [enterprise](), [hadoop](), [kubernetes](), [spark]() ### AI overview This case study explains why Anaconda selected Teleport's Gravity to package and distribute its Kubernetes-based Anaconda Enterprise application for enterprise customers. It describes Anaconda's need to support cloud, hybrid, and on-premises environments while meeting security and compliance requirements. ### Source excerpt Learn why Anaconda selected Teleport to package their Kubernetes-based application to Enterprise customers. ## Even Faster ORC DevFeed: [Even Faster ORC]() Original publisher: [Read original article]() Author: Dain Sundstrom, Martin Traverso Published: 2019-04-23T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Optimization](), [Code](), [Hadoop](), [SQL](), [Benchmark](), [Programming]() Tags: [batch](), [benchmark](), [code](), [cpu](), [hadoop](), [optimization](), [performance](), [sql]() ### AI overview This article explains how Trino improved its custom ORC reader. The changes optimize type-specific decoding, bulk reads, null handling, and dynamic dispatch, reducing TPC-DS query time by about 5% and CPU usage by about 9%. ### Source excerpt Trino is known for being the fastest SQL on Hadoop engine, and our custom ORC reader implementation is a big reason for this speed - now it is even faster! ## How to Setup a Scheduled Scala Spark Job DevFeed: [How to Setup a Scheduled Scala Spark Job]() Original publisher: [Read original article]() Published: 2019-03-27T00:00:00Z Content type: tutorial Language: en Sources: [Curalate]() Topics: [Apache Spark](), [Scala](), [Amazon Web Services](), [Amazon S3](), [Hadoop](), [SDKs](), [Command-line interface]() Tags: [aws](), [cli](), [daily](), [data](), [ec2](), [emr](), [github](), [hadoop](), [how-to](), [job](), [pipeline](), [s3](), [scala](), [scheduled](), [sdk](), [spark](), [weekly]() ### AI overview A tutorial for packaging and deploying a Scala Spark job as a fat JAR, uploading it to Amazon S3, and configuring it to run on a schedule through AWS Data Pipeline and EMR. ### Source excerpt Have you written a Scala Spark job that processes a massive amount of data on an intimidating amount of RAM and you want to run it daily/weekly/monthly on a schedule on AWS? I had to do this recently, and couldn't find a good tutorial on the full process to get the spark job running. Included in this article and accompanying repository is everything you need to get your Scala Spark job running on AWS Data Pipeline and EMR. Code Repo This tutorial is not going to walk you through the process of actually writing your specific Scala Spark job to do whatever number crunching you need. There are already plenty of resources available (1, 2, 3) to get you started on that. The code template for setting up a Spark Scala job is available in this GitHub repo. Assuming that you have already written your Spark Job and are only using the AWS Java SDK to connect to your AWS data stores, drop your code in the Main function of SparkJob.scala and run the deploy.sh script to upload the fat jar to your S3 bucket. If you do take other dependencies, then it may take some extra work on your part. To run a Scala Spark job on AWS you need to compile a fat jar that contains the byte code for your job and all of the libraries it needs to run. This project already has the sbt-assembly plugin setup and a assemblyMergeStrategy set up to package the Spark, Hadoop, and AWS SDK together in the fat jar. If you need to add in other libraries that do not play well with each other, or are using a noncompatible version of Spark for this current repo, there are a few good resources available to help you through the needed build.sbt modifications. Outside of the previously mentioned needed changes you need to set a few parameters in the deploy.sh script. Mainly the deploymentPath to your specific S3 bucket, adding a profile to the AWS CLI command to upload to your specific S3 bucket if it's private, and changing the resulting fat jar name if you please. The deploy script uses the AWS CLI to upload the fat ## Announcing OpenTSDB 2.4.0: Rollup and Pre-Aggregation Storage, Histograms, Sketches, and More DevFeed: [Announcing OpenTSDB 2.4.0: Rollup and Pre-Aggregation Storage, Histograms, Sketches, and More]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-12-27T17:01:17Z Content type: release Language: en Sources: [Yahoo]() Topics: [Databases](), [Time Series](), [Hadoop](), [Monitoring](), [Data analysis](), [Open Source](), [Temporal data]() Tags: [data-analysis](), [databases](), [hadoop](), [monitoring](), [open-source](), [release](), [salesforce](), [time-series](), [workflows](), [yahoo-engineering]() ### AI overview OpenTSDB 2.4.0 is released with rollup and pre-aggregation storage for time series data, along with histograms and sketches. The release focuses on retaining lower-resolution data for longer periods, reducing query workloads, and supporting more accurate percentile analysis across multiple series. ### Source excerpt yahoodevelopers: By Chris Larsen, Architect OpenTSDB is one of the first dedicated open source time series databases built on top of Apache HBase and the Hadoop Distributed File System. Today, we are proud to share that version 2.4.0 is now available and has many new features developed in-house and with contributions from the open source community. This release would not have been possible without support from our monitoring team, the Hadoop and HBase developers, as well as contributors from other companies like Salesforce, Alibaba, JD.com, Arista and more. Thank you to everyone who contributed to this release! A few of the exciting new features include: Rollup and Pre-Aggregation Storage As time series data grows, storing the original measurements becomes expensive. Particularly in the case of monitoring workflows, users rarely care about last years' high fidelity data. It's more efficient to store lower resolution "rollups" for longer periods, discarding the original high-resolution data. OpenTSDB now supports storing and querying such data so that the raw data can expire from HBase or Bigtable, and the rollups can stick around longer. Querying for long time ranges will read from the lower resolution data, fetching fewer data points and speeding up queries. Likewise, when a user wants to query tens of thousands of time series grouped by, for example, data centers, the TSD will have to fetch and process a significant amount of data, making queries painfully slow. To improve query speed, pre-aggregated data can be stored and queried to fetch much less data at query time, while still retaining the raw data. We have an Apache Storm pipeline that computes these rollups and pre-aggregates, and we intend to open source that code in 2019. For more details, please visit http://opentsdb.net/docs/build/html/user_guide/rollups.html. Histograms and Sketches When monitoring or performing data analysis, users often like to explore percentiles of their measurements, such as the 9 ## Omid 1.0.0 adds low-latency transaction processing for Apache Phoenix DevFeed: [Omid 1.0.0 adds low-latency transaction processing for Apache Phoenix]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-12-06T19:32:04Z Content type: release Language: en Sources: [Yahoo]() Topics: [phoenix](), [Transactions](), [Hadoop](), [SQL](), [Deployment](), [Cloud]() Tags: [big-data](), [cloud](), [deployment](), [hadoop](), [low-latency](), [open-source](), [phoenix](), [real-time](), [speed](), [sql](), [transactions](), [yahoo-engineering]() ### AI overview This release article introduces Omid 1.0.0, an open source transaction processing platform for Big Data, and announces its selection as the transaction management provider for Apache Phoenix. It describes the Omid Low Latency protocol, which reduces short-transaction latency by 5 times under light load and by 10 to 100 times under heavy load. ### Source excerpt yahoodevelopers: By Ohad Shacham, Yonatan Gottesman, Edward Bortnikov Scalable Systems Research, Verizon/Oath Omid, an open source transaction processing platform for Big Data, was born as a research project at Yahoo (now part of Verizon), and became an Apache Incubator project in 2015. Omid complements Apache HBase, a distributed key-value store in Apache Hadoop suite, with a capability to clip multiple operations into logically indivisible (atomic) units named transactions. This programming model has been extremely popular since the dawn of SQL databases, and has more recently become indispensable in the NoSQL world. For example, it is the centerpiece for dynamic content indexing of search and media products at Verizon, powering a web-scale content management platform since 2015. Today, we are excited to share a new chapter in Omid's history. Thanks to its scalability, reliability, and speed, Omid has been selected as transaction management provider for Apache Phoenix, a real-time converged OLTP and analytics platform for Hadoop. Phoenix provides a standard SQL interface to HBase key-value storage, which is much simpler and in many cases more performant than the native HBase API. With Phoenix, big data and machine learning developers get the best of all worlds: increased productivity coupled with high scalability. Phoenix is designed to scale to 10,000 query processing nodes in one instance and is expected to process hundreds of thousands or even millions of transactions per second (tps). It is widely used in the industry, including by Alibaba, Bloomberg, PubMatic, Salesforce, Sogou and many others. We have just released a new and significantly improved version of Omid (1.0.0), the first major release since its original launch. We have extended the system with multiple functional and performance features to power a modern SQL database technology, ready for deployment on both private and public cloud platforms. A few of the significant innovations include: Protocol ## Hadoop Contributors Meetup at Oath DevFeed: [Hadoop Contributors Meetup at Oath]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-11-08T19:07:25Z Content type: news Language: en Sources: [Yahoo]() Topics: [Hadoop](), [Development](), [Containers](), [Latency](), [Open Source]() Tags: [big-data](), [containers](), [development](), [docker](), [hadoop](), [low-latency](), [open-source](), [yahoo-engineering]() ### AI overview Oath hosted a day-long Hadoop Contributors Meetup at its Sunnyvale campus, bringing together more than 80 Hadoop users, contributors, committers, and PMC members. Talks and breakout sessions covered challenges and solutions across the Hadoop ecosystem, including HDFS, Tez, containers, low-latency processing, deep learning, scalability, security, and cloud deployment. ### Source excerpt yahoodevelopers: By Scott Bush, Director, Hadoop Software Engineering, Oath On Tuesday, September 25, we hosted a special day-long Hadoop Contributors Meetup at our Sunnyvale, California campus. Much of the early Hadoop development work started at Yahoo, now part of Oath, and has continued over the past decade. Our campus was the perfect setting for this meetup, as we continue to make Hadoop a priority. More than 80 Hadoop users, contributors, committers, and PMC members gathered to hear talks on key issues facing the Hadoop user community. Speakers from Ampool, Cloudera, Hortonworks, Microsoft, Oath, and Twitter detailed some of the challenges and solutions pertinent to their parts of the Hadoop ecosystem. The talks were followed by a number of parallel, birds of a feather breakout sessions to discuss HDFS, Tez, containers and low latency processing. The day ended with a reception and consensus that the event went well and should be repeated in the near future. Presentation recordings (YouTube playlist) and slides (links included in the video description) are available here: Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda Tan, Hortonworks Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara, Botong Huang "HDFS Scalability and Security", Daryn Sharp, Senior Engineer, Oath The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, Ampool Moving the Oath Grid to Docker, Eric Badger, Software Developer Engineer, Oath Vespa: Open Source Big Data Serving Engine, Jon Bratseth, Distinguished Architect, Oath Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shane Kumpf, Hortonworks How Twitter Hadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenu Thank you to all the presenters and the attendees both in person and remote! P.S. We're hiring! Learn more about career opportunities at Oath. [Next page]()