# hdfs Published articles for hdfs. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Inside Cilium CNI: solving mysterious Kubernetes pod setup timeouts DevFeed: [Inside Cilium CNI: solving mysterious Kubernetes pod setup timeouts]() Original publisher: [Read original article]() Author: Adyen Published: 2026-07-13T09:17:40Z Content type: article Language: en Sources: [Adyen Tech]() Topics: [Cilium](), [Kubernetes](), [Kubernetes networking](), [networking](), [eBPF](), [Latency](), [hdfs](), [Apache Spark](), [big-data](), [cpu]() Tags: [big-data](), [cilium](), [cni](), [cpu](), [data](), [devops](), [ebpf](), [hdfs](), [kubernetes](), [kubernetes-networking](), [latency](), [spark](), [speed](), [tracing]() ### AI overview An Adyen engineer investigates Cilium CNI timeouts that prevented new Kubernetes pods from starting. The article attributes the issue to sequential connection-tracking-table garbage collection, which became a bottleneck under workloads involving HDFS, Trino, and short-lived Spark jobs. ### Source excerpt By Jorrick Sleijster - Senior Data Platform Engineer, Adyen I was fully aware a year ago that a single configuration line could break the Kubernetes networking stack. But if they told me that leftovers from Kubernetes pods which terminated hours prior could block new ones from starting, I would have thought they were joking. In high-performance networking, 35 seconds is a lifetime. This was the latency required to iterate through our connection tracking table of 7 million entries at a maximum speed of 200,000 entries per second. At our 16-million-entry peak, this sequential lookup could take up to 80 seconds, leading to Cilium CNI timeouts preventing new pods from starting on affected nodes. We uncovered this linear-time behavior at Adyen by tracing syscalls, inspecting codebases, and analyzing eBPF internals. This investigation revealed how our varied workloads turned the connection tracking table's garbage collection algorithm into a critical bottleneck. Our setup: why we're different At Adyen, we run Cilium CNI across all our 100+ Kubernetes clusters. When we switched from Calico to Cilium, we knew we'd face challenges adapting it to our production workloads. Our production big data Kubernetes clusters have a unique usage pattern compared to the other Kubernetes environments within Adyen: Data extraction from HDFS. Our infrastructure relies on more than 500 datanodes. Trino represents one of our most demanding HDFS workloads, processing analytical queries against data stored on HDFS. Due to the distributed nature of HDFS, each file you download requires a new connection to any of these 500 nodes. Therefore, during peak hours, a single pod can produce approximately 50,000 connections every minute. Pod churn. Many pods we spawn on the Kubernetes cluster run batch jobs, such as Spark jobs. They stay around for anywhere from a second to a couple of hours. Wide variety of workloads. Some workloads are very CPU-intensive, like Spark pods executing complex joins and tra ## Why HTAP Databases Are Giving Way to Disaggregated Architectures DevFeed: [Why HTAP Databases Are Giving Way to Disaggregated Architectures]() Original publisher: [Read original article]() Author: Zhou Sun Published: 2025-05-04T10:00:00Z Content type: article Language: en Sources: [Blog -- Neon Docs]() Topics: [Databases](), [olap](), [Cloud](), [Latency](), [consistency](), [NoSQL](), [MongoDB](), [Hadoop](), [hdfs](), [CockroachDB](), [Amazon Redshift](), [vitess]() Tags: [cloud](), [cloud-data](), [cockroachdb](), [consistency](), [data-lake](), [databases](), [engineering](), [hadoop](), [hdfs](), [latency](), [mongodb](), [olap](), [redshift](), [sql](), [vitess]() ### AI overview The article traces the separation of transactional and analytical database workloads, explaining how differing storage and scaling requirements led to specialized OLTP and OLAP systems. It argues that HTAP as a single database architecture is declining while its underlying ideas persist in today's disaggregated data stack. ### Source excerpt This blog is inspired by Jordan Tigani's "Big Data is Dead." Jordan and I actually spent some time building an HTAP database at SingleStore. From the one database that did everything in the '80s, to the great divide, to HTAP, to today's disaggregated stack--here's why HTAP as a database is dead, but its spirit lives on. ## How Uber Reduced Their Log Size By 99% DevFeed: [How Uber Reduced Their Log Size By 99%]() Original publisher: [Read original article]() Author: Richard Oliver Bray Published: 2024-10-09T13:02:55Z Content type: article Language: en Sources: [Hacking Scale by Better Stack]() Topics: [Logging](), [Hadoop](), [Filesystems](), [big-data](), [Apache Spark](), [data]() Tags: [apache-spark](), [big-data](), [cli](), [clickhouse](), [data](), [hadoop](), [hdfs](), [logging]() ### AI overview The article explains how Uber addressed the storage cost of generating roughly 5 PB of INFO-level logs each month. It describes Uber's use of HDFS and related data-processing tools, while reporting that the company reduced log storage size by 99%. ### Source excerpt Uber broke apart an open source tool to massively compress their logs ## Implementing Data Validation with Great Expectations in Hybrid Environments DevFeed: [Implementing Data Validation with Great Expectations in Hybrid Environments]() Original publisher: [Read original article]() Author: Kamila Widyanto Full time DevOps; Site Reliability Engineer; Part Time Rubberduck Linkedin Profile Published: 2023-04-25T00:00:00Z Content type: tutorial Language: en Sources: [Trivago]() Topics: [data-processing](), [Hadoop](), [hdfs](), [integrity](), [Apache Spark](), [configuration](), [Library](), [Python](), [JSON](), [Open Source](), [YAML](), [version-control]() Tags: [configuration](), [data](), [data-pipeline](), [data-processing](), [data-validation](), [devops](), [distributed](), [hadoop](), [hdfs](), [integrity](), [json](), [library](), [open-source](), [python](), [version-control](), [workflow](), [yaml]() ### AI overview This article describes implementing Great Expectations for data validation in a hybrid Hadoop environment. It explains the framework's core concepts and how the authors ran it as a PySpark job in an automated data pipeline, including configuring the Data Context for HDFS constraints. ### Source excerpt Data validation is an essential step in any data processing pipeline, as it ensures the integrity and accuracy of the data to be used across all subsequent processing steps. ## Journey to Iceberg with Trino DevFeed: [Journey to Iceberg with Trino]() Original publisher: [Read original article]() Author: JaeChang Song, Jennifer Oh, Brian Olsen Published: 2022-12-19T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Apache Iceberg](), [data-engineering](), [big-data](), [dashboards](), [Hadoop]() Tags: [data-engineering](), [engineering](), [hadoop](), [hdfs](), [logs](), [metrics](), [open-source](), [performance](), [post](), [scale](), [speed](), [summit](), [switching]() ### AI overview SK Telecom describes its journey from Hive-based Trino deployments to Iceberg after encountering scaling and performance problems. The company used Trino across Hadoop and HDFS-based data platforms, collected query plans, JMX statistics, system metrics, and logs, and built a dashboard to investigate blocked queries and cluster behavior. ### Source excerpt This post comes from the second half of Trino Summit 2022 session. Our friends JaeChang and Jennifer from SK Telecom traveled across the globe from South Korea to join us in person! SK Telecom recently had some issues scaling Trino on the Hive model, among other issues that come with Hive. While some initial tweaking helped speed things up, it ultimately never solved the problem. After switching to Iceberg, SK Telecom ran initial performance tests with some very impressive results. In this talk, Jennifer and JaeChang describe their journey to Iceberg with Trino. ## Pinion -- The Load Framework Part-2 DevFeed: [Pinion -- The Load Framework Part-2]() Original publisher: [Read original article]() Author: Saurabh Jain Published: 2021-10-29T16:50:24Z Content type: article Language: en Sources: [Groupon Engineering -- Medium]() Topics: [data lake](), [Framework](), [Data Quality](), [Transactions](), [Streaming](), [Open Source]() Tags: [acid](), [audit](), [aws](), [cloud](), [data-validation](), [delta-lake](), [deltalake](), [hdfs](), [logging](), [optimization](), [reporting](), [s3](), [schema](), [science](), [spark](), [streaming](), [transactions]() ### AI overview This second post in the Pinion -- The Load Framework series explains how Pinion extends Apache Delta Lake APIs for slowly changing dimension operations. It covers data validation, compaction, auditing, streamlined logging, and chained APIs, and introduces Delta Lake capabilities such as ACID transactions, schema enforcement, batch and streaming interfaces, and time travel. ### Source excerpt Pinion -- The Load Framework Part-2 This post is the 2nd part of the "Pinion -- The Load Framework" series. In case you have not read the 1st post, you can read it here. In this post, we are going to cover the following topics. How does Pinion use Delta Lake for SCD operations? Small file problem with Delta Lake and its resolution. Before we dive into the topics of this post, let's look at the definition of DeltaLake to set the context right. Apache Delta Lake - Apache Delta Lake is an open-source framework that enables the addition of ACID transactions support to a new data lake or an existing data lake created on top of S3, GCS, and HDFS. In addition to this, it provides other features such as scalable metadata handling, unified interface for both batch and streaming application, schema enforcement, time travel, and a rich interface of APIs to enable complex use cases like change-data-capture (CDC) and slowly-changing-dimension (SCD) operations. To keep the post concise and to the point, I won't go into much detail here about Delta Lake, since there is already great documentation available about it, that you can read it here. How does Pinion use Delta Lake for SCD operations? - Apache Delta Lake provides a rich set of APIs to handle slowly-changing dimensions, however, those APIs were not enough alone to build the features that we want to have in The Pinion Framework. So, we decided to enrich the APIs provided by Delta Lake by adding the following features to it: Data Validation Compaction Audit Streamlined logging infrastructure to make the data engineer's life easier during debugging of a failed job Chained APIs Let's dive a little further into the features that we had listed above. Data Validation -- By default, schema enforcement is enabled in Pinion for all the APIs where we have a need of inserting rows from source data(LRFs) into the target table. In case of a schema mismatch, Pinion raises an error and stops processing of further stages. It ensures the data a ## 分布式文件系统的演化 DevFeed: [分布式文件系统的演化]() Original publisher: [Read original article]() Author: Joway Published: 2020-06-14T00:00:00Z Content type: tutorial Language: zh Sources: [Random Thoughts]() Topics: [IO](), [Linux](), [Ext4](), [hdfs](), [Google]() Tags: [blocks](), [client](), [ext4](), [filesystem](), [google](), [hdfs](), [io](), [lease](), [linux](), [master](), [pointers](), [primary](), [ssd](), [tech]() ### AI overview This article explains local filesystem structure, including superblocks, inodes, block pointers, and file-size limits, then introduces the evolution and architecture of distributed filesystems. It uses GFS to describe metadata lookup, chunk locations, caching, replication, leases, and client read and write flows, and mentions HDFS as a related implementation. ### Source excerpt 文件系统是操作系统 IO 栈里非常重要的一个中间层，其存在的意义是为了让上层应用程序有一层更加符合人类直觉的抽象来进行文档的读写，而无需考虑底层存储上的细节。 本地文件系统 在讨论分布式文件系统前，我们先来回顾下本地文件系统的组成。 存储结构 在前面一张图里，我们能够看到文件系统直接和通用块层进行交互，无论底层存储介质是磁盘还是 SSD，都被该层抽象为 Block 的概念。文件系统在初始化时，会先在挂载的块存储上的第一个位置创建一个 Super Block： ## Hive 3 support in Presto DevFeed: [Hive 3 support in Presto]() Original publisher: [Read original article]() Author: Piotr Findeisen, Starburst Data Published: 2019-12-28T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Hadoop](), [data](), [ci](), [Availability]() Tags: [compatibility](), [continuous-integration](), [hadoop](), [hdfs](), [integration]() ### AI overview This article summarizes Presto's compatibility with Hive 3, including support for Hadoop Erasure Coding, transactional tables, timestamp values stored in ORC, and Hive bucketing v2. It also describes compatibility improvements delivered across Presto releases and ongoing work toward fuller Hive 3 integration. ### Source excerpt The Hive community is centered around a few different Hive distributions, one of them being Hortonworks Data Platform (HDP). Even after the Cloudera-Hortonworks merger there is vivid interest in HDP 3, featuring Hive 3. Presto is ready for the game. In this post, we summarize which Hive 3 features Presto already supports, covering all the work that went into Presto to achieve that. We also outline next steps lying ahead. ## Linux I/O 栈浅析 DevFeed: [Linux I/O 栈浅析]() Original publisher: [Read original article]() Author: Joway Published: 2019-08-11T00:00:00Z Content type: tutorial Language: zh Sources: [Random Thoughts]() Topics: [IO](), [Linux](), [Cache](), [file](), [Ext4](), [XFS]() Tags: [block](), [cache](), [ceph](), [ext4](), [file](), [filesystem](), [hdd](), [hdfs](), [io](), [linux](), [mysql](), [oracle](), [ssd](), [tech](), [xfs]() ### AI overview This article provides a high-level overview of Linux file I/O, tracing the path from applications through VFS, page cache, filesystems, FUSE, block devices, and I/O scheduling. It also explains hard and symbolic links, direct I/O, filesystem partitioning, and access to raw devices. ### Source excerpt 在 Linux 中，所有外部资源都以文件形式作为一个抽象视图，并提供一套统一的接口给应用程序调用。本文将以宏观视角试图阐述 Linux 中关于文件 IO 的整个调用脉络。 VFS 在 Linux 中，所有 IO 都必须先经由 VFS 层进行转发。通过 VFS 将包括磁盘、网络 Socket、打印机、管道等资源全部封装成统一的接口。 ## Success at Apache: Migrating Yahoo Jobs from Apache MapReduce to Apache Tez DevFeed: [Success at Apache: Migrating Yahoo Jobs from Apache MapReduce to Apache Tez]() Original publisher: [Read original article]() Author: mikesefanov Published: 2018-02-05T14:43:07Z Content type: article Language: en Sources: [Yahoo]() Topics: [Hadoop](), [Open Source](), [data](), [Apache Spark]() Tags: [2017](), [apache](), [blog](), [bug](), [data](), [distributed](), [hadoop](), [hdfs](), [oath](), [open-source](), [spark](), [tez](), [yahoo](), [yahoo-engineering]() ### AI overview Kuhu Shukla describes her experience contributing to Apache projects at Yahoo and migrating Yahoo jobs from Apache MapReduce to Apache Tez. The article also discusses Oath's Apache-based grid infrastructure and her early work with YARN, HDFS, and Hadoop. ### Source excerpt yahoodevelopers: Kuhu Shukla (bottom center) and team at the 2017 DataWorks Summit By Kuhu Shukla This post first appeared here on the Apache Software Foundation blog as part of ASF's "Success at Apache" monthly blog series. As I sit at my desk on a rather frosty morning with my coffee, looking up new JIRAs from the previous day in the Apache Tez project, I feel rather pleased. The latest community release vote is complete, the bug fixes that we so badly needed are in and the new release that we tested out internally on our many thousand strong cluster is looking good. Today I am looking at a new stack trace from a different Apache project process and it is hard to miss how much of the exceptional code I get to look at every day comes from people all around the globe. A contributor leaves a JIRA comment before he goes on to pick up his kid from soccer practice while someone else wakes up to find that her effort on a bug fix for the past two months has finally come to fruition through a binding +1. Yahoo - which joined AOL, HuffPost, Tumblr, Engadget, and many more brands to form the Verizon subsidiary Oath last year - has been at the frontier of open source adoption and contribution since before I was in high school. So while I have no historical trajectories to share, I do have a story on how I found myself in an epic journey of migrating all of Yahoo jobs from Apache MapReduce to Apache Tez, a then-new DAG based execution engine. Oath grid infrastructure is through and through driven by Apache technologies be it storage through HDFS, resource management through YARN, job execution frameworks with Tez and user interface engines such as Hive, Hue, Pig, Sqoop, Spark, Storm. Our grid solution is specifically tailored to Oath's business-critical data pipeline needs using the polymorphic technologies hosted, developed and maintained by the Apache community. On the third day of my job at Yahoo in 2015, I received a YouTube link on An Introduction to Apache Tez. I watched ## Running Apache Flink on Amazon Elastic Mapreduce DevFeed: [Running Apache Flink on Amazon Elastic Mapreduce]() Original publisher: [Read original article]() Published: 2016-01-06T08:15:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [apache-flink](), [Amazon Web Services](), [Hadoop](), [Scalding](), [data-processing](), [Amazon S3](), [Apache Spark]() Tags: [apache-flink](), [aws](), [data-processing](), [ec2](), [emr](), [flink](), [hadoop](), [hdfs](), [linux](), [realtime](), [s3](), [scala](), [scalding](), [sclading](), [spark](), [streaming]() ### AI overview A tutorial on running Apache Flink jobs on Amazon EMR, including adapting Scalding jobs and launching and accessing an EMR cluster. It also discusses EMRFS support for reading data from Amazon S3. ### Source excerpt I love really Amazon EMR. Over the years it's grown from being "Hadoop on-demand" to a full-fledged cluster management system for running OSS big-data apps (Hadoop MR of course, but also Spark, Hue, Hive, Pig, Oozie and more). While Hadoop out of the box supports reading from S3, EMR has a proprietary implementation called EMRFS that has some nice features. For those reasons, it's really the best Hadoop cluster to use if you're storing your data in S3. Lately I've been experimenting a lot with Apache Flink to replace MR as the excution fabric. At work, we have many, many jobs written in Scalding. Flink can execute Scalding jobs with some very simple modifications which was a great way to move our jobs from MR to a more memory-centric data processing model. However, we really wanted to run our jobs on EMR using Flink. Flink is not an option for EMR (yet) but can we still get our jobs to run? Let's see! Start an EMR cluster The first thing we need is an EMR cluster. You can launch a small test cluster for very cheap. Once it's running, let's ssh onto it and see what's going on. ihummel at mm-mac-3270 in ~ $ ssh hadoop@ec2-54-226-25-85.compute-1.amazonaws.com Last login: Wed Jan 6 16:33:01 2016 from 172.85.47.138 __| __|_ ) _| ( / Amazon Linux AMI ___|\___|___| https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/ 23 package(s) needed for security, out of 49 available Run "sudo yum update" to apply all updates. EEEEEEEEEEEEEEEEEEEE MMMMMMMM MMMMMMMM RRRRRRRRRRRRRRR E::::::::::::::::::E M:::::::M M:::::::M R::::::::::::::R EE:::::EEEEEEEEE:::E M::::::::M M::::::::M R:::::RRRRRR:::::R E::::E EEEEE M:::::::::M M:::::::::M RR::::R R::::R E::::E M::::::M:::M M:::M::::::M R:::R R::::R E:::::EEEEEEEEEE M:::::M M:::M M:::M M:::::M R:::RRRRRR:::::R E::::::::::::::E M:::::M M:::M:::M M:::::M R:::::::::::RR E:::::EEEEEEEEEE M:::::M M:::::M M:::::M R:::RRRRRR::::R E::::E M:::::M M:::M M:::::M R:::R R::::R E::::E EEEEE M:::::M MMM M:::::M R:::R R::::R EE:::::EEEEEEEE::::E ## Getting started with Scalding DevFeed: [Getting started with Scalding]() Original publisher: [Read original article]() Published: 2015-12-20T08:15:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [Scalding](), [data-processing](), [Hadoop](), [Scala](), [Testing](), [Compression](), [Logging]() Tags: [beginners](), [build](), [building](), [compression](), [getting-started](), [hadoop](), [hdfs](), [logging](), [scala](), [scalding](), [sclading](), [testing](), [unit-testing]() ### AI overview A practical guide to starting Scalding projects with SBT. It presents a minimal project structure and dependencies, explains local Hadoop execution and logging, and shows how to test jobs locally before submitting them to a cluster. ### Source excerpt I've been using Scalding for the last few years and really love how simple it makes writing scalalbe data processing jobs. I think many of the issues beginners have with Scalding relate to project setup. I hope this post simplifies things for people so they can started with less hassle. Building your project with SBT The official getting started guide is pretty terse. For our purposes, all you need to get started is a very simple SBT project. simple-scalding-example/ |-- build.sbt |-- project | |-- build.properties | `-- plugins.sbt |-- src | `-- main | `-- scala | `-- WordCountJob.scala `-- log4j.properties The most important thing the setup of the build.sbt file. Our example has minimal dependencies: libraryDependencies ++= Seq( "com.twitter" %% "scalding-core" % "0.15.0", "org.apache.hadoop" % "hadoop-client" % "2.2.0" % "provided", "org.slf4j" % "slf4j-log4j12" % "1.7.13" % "provided" ) The hadoop-client jar is included for compilation, unit testing and running locally on your laptop but will not be pacakged into the final "fat jar" assembly shipped out to the cluster. Additionally, a logging framework is included so that we get good logs when debugging locally. A simple log4j.properties file is included in the repo. The whole job can be run locally using real Hadoop libraries without any additional software install: $ sbt 'run net.themodernlife.WordCount --hdfs --input build.sbt --output target/output' It's even possible to get things like LZO-compression to work with this setup. The main win here is that there is a very simple, repeatable development environment that can be used to test all aspects of the job locally before sending things off to the cluster. ## Running Scalding jobs on Apache Flink DevFeed: [Running Scalding jobs on Apache Flink]() Original publisher: [Read original article]() Published: 2015-12-20T08:15:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [Scalding](), [apache-flink](), [Scala](), [Streaming](), [Hadoop]() Tags: [apache-flink](), [flink](), [hadoop](), [hdfs](), [realtime](), [scala](), [scalding](), [sclading](), [streaming]() ### AI overview This tutorial explains how to run Scalding jobs on Apache Flink. It outlines the required compatibility work, including Cascading 3 support, a Scalding version compiled against Cascading 3, selection of hadoop2-flink as the Cascading fabric, and a fix for a Twitter Chill bug. It also provides a sample Scala word-count job and build configuration changes. ### Source excerpt My previous post showed a very simple Scalding workflow. Apache Flink is a real time streaming framework that's very promising. It also supports running Cascading workflows with very little modification. Surely there must be some way to run a Scalding job on top of Flink? Turns out... YES! In a nutshell Here are the high-level things we need to solve for We need a Scalding job to test this out with cascading-flink requires Cascading 3 We need a new version of Scalding - Compiled against Cascading 3 - Allows hadoop2-flink to be selected as the Cascading "fabric" There is a bug in Twitter Chill We need Flink packaged with the Chill fix Scalding job Let's start with a very simple Scalding job. You can download https://github.com/themodernlife/simple-scalding-example for some inspiration. package net.themodernlife import com.twitter.scalding._ class WordCount(args: Args) extends Job(args) { def tokenize(text: String): Array[String] = { text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("\\s+") } val input = args("input") val output = args("output") TextLine(args("input")) .flatMap[String, String]('line -> 'word)(tokenize) .groupBy('word)(_.size) .write(Tsv(output)) } We'll be making some updates to our build.sbt. Here's what we're starting with organization := "net.themodernlife" name := "simple-scalding-example" scalaVersion := "2.11.7" scalacOptions ++= Seq("-encoding", "utf-8", "-deprecation", "-unchecked", "-feature") resolvers ++= Seq( "Concurrent Maven Repo" at "http://conjars.org/repo", "Twitter Maven Repo" at "http://maven.twttr.com" ) libraryDependencies ++= Seq( "com.twitter" %% "scalding-core" % "0.15.0", "org.apache.hadoop" % "hadoop-client" % "2.2.0" % "provided", "org.slf4j" % "slf4j-log4j12" % "1.7.13" % "provided" ) A new Scalding build We need to update Scalding Apply https://github.com/twitter/scalding/pull/1446 Apply/hack https://github.com/twitter/scalding/pull/1220 Hack Build.scala to ignore some submodules According to http://www.cascading.org/2 ## ApacheCon Budapest 2014 DevFeed: [ApacheCon Budapest 2014]() Original publisher: [Read original article]() Author: mick Published: 2014-11-25T13:00:00Z Content type: opinion Language: en Sources: [Finn.no]() Topics: [Hadoop](), [Apache Spark](), [Docker](), [hdfs](), [Apache Cassandra](), [Streaming](), [couchdb](), [Data structures]() Tags: [apache](), [cassandra](), [couchdb](), [data-structures](), [docker](), [hadoop](), [hdfs](), [spark](), [streaming]() ### AI overview A firsthand account of ApacheCon Budapest 2014 discusses Apache communities and several technologies, including Docker, Spark, Cassandra, CouchDB, Hadoop, Mesos, Aurora, YARN, Solr, and OpenOffice. It highlights considerations around scalability, testing, streaming, production utilisation, and data processing. ### Source excerpt Last week was Europe's ApacheCon, held in Budapest. A refreshingly "laid-back and no-nonsense" conference free from the douchebaggery that comes from big sponsors and marketing. This year it was held in the awesome 5-star Corinthia Hotel, and consisted of a welcoming and cheerful vibe, a place for apache members, committers, and all, to get together and better know each other. Apache continues to grow as the foundation for strong communities building trusted open sourced solutions, for running half the internet, and providing a substantial amount of code to your java stack. Some hot topics were Docker, Mesos, Spark, Cassandra, CouchDB, Hadoop, Solr, OpenOffice, and the need for greater diversity in our communities. Spark is all the rage because of its brevity and simplicity, but isn't really a complete solution yet because it doesn't scale in many situations. Docker is awesome, and looks to take over the testing domain, but it remains limited and out of production as long as it has no network stack implementation. There was a very cool demonstration of Mesos and Aurora increasing production utilisation, even allowing in quiet periods for development/testing servers to come in. The next release of YARN (2.6) will see support for long-lived services, whereby we can transform many of our hadoop jobs with very little effort into true streaming solutions. YARN will also be able to deploy docker containers through its clusters, this could be a very nice solution for our batch jobs. Putting these together and you see that the technical separation between streaming and aggregating solutions fades away and it really just boils down to what data structures you solve each use case with. Talking with a hadoop committer it came to light that FINN could be running the fastest hadoop cluster that they know of, given HDFS runs on SSDs and is dedicated just for hadoop internals. Of no surprise Solr dealt a lot with scaling and performance, while Cassandra presented use-cases from ac ## Getting the current filename with Spark and HDFS DevFeed: [Getting the current filename with Spark and HDFS]() Original publisher: [Read original article]() Published: 2014-09-28T11:50:13Z Content type: tutorial Language: en Sources: [Ian Hummel]() Topics: [Apache Spark](), [Hadoop]() Tags: [apache-spark](), [foreach](), [hadoop](), [hdfs](), [map](), [reduce](), [scala](), [spark](), [subclass]() ### AI overview A tutorial shows how to obtain the filename being processed in a Spark map/reduce job. It uses Spark 1.1.0's HadoopRDD.mapPartitionsWithInputSplit with FileSplit, and notes that the approach requires an InputFormat subclass of FileInputFormat. ### Source excerpt It's occasionally useful when writing map/reduce jobs to get a hold of the current filename that's being processed. There's a few ways to do this, depending on the version of Spark that you're using. Spark 1.1.0 introduced a new method on HadoopRDD that makes this super easy: import org.apache.hadoop.io.LongWritable import org.apache.hadoop.io.Text import org.apache.hadoop.mapred.{FileSplit, TextInputFormat} import org.apache.spark.rdd.HadoopRDD // Create the text file val text = sc.hadoopFile("file:///tmp/files", classOf[TextInputFormat], classOf[LongWritable], classOf[Text], sc.defaultMinPartitions) // text: org.apache.spark.rdd.RDD[(org.apache.hadoop.io.LongWritable, org.apache.hadoop.io.Text)] = file:///tmp/files HadoopRDD[2] at hadoopFile at :16 // Cast to a HadoopRDD val hadoopRdd = text.asInstanceOf[HadoopRDD[LongWritable, Text]] // hadoopRdd: org.apache.spark.rdd.HadoopRDD[org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text] = file:///tmp/files HadoopRDD[2] at hadoopFile at :16 val fileAndLine = hadoopRdd.mapPartitionsWithInputSplit { (inputSplit, iterator) ⇒ val file = inputSplit.asInstanceOf[FileSplit] iterator.map { tpl ⇒ (file.getPath, tpl._2) } } fileAndLine.foreach(println) // prints lines like (file:/tmp/files/part-00000,line1...), (file:/tmp/files/part-00000,line2...), ... There are a few things to note: - You can't simply use sc.textFile because under the hood, the textFile method actually maps the resulting HadoopRDD, returning a MappedRDD which is not what we want - This will only work if your InputFormat is a subclass of FileInputFormat - mapPartitionsWithInputSplit returns a new iterator, in this case we simply map over it (which is done lazily!) to include the split's filename