# hdfs HDFS is the Hadoop Distributed File System, a distributed file system designed to store very large files across machine clusters. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Inside Cilium CNI: solving mysterious Kubernetes pod setup timeouts DevFeed: [Inside Cilium CNI: solving mysterious Kubernetes pod setup timeouts]() Original publisher: [Read original article]() Author: Adyen Published: 2026-07-13T09:17:40Z Content type: article Language: en Sources: [Adyen Tech]() Topics: [Cilium](), [Kubernetes](), [Kubernetes networking](), [networking](), [eBPF](), [Latency](), [hdfs](), [Apache Spark](), [big-data](), [cpu]() Tags: [big-data](), [cilium](), [cni](), [cpu](), [data](), [devops](), [ebpf](), [hdfs](), [kubernetes](), [kubernetes-networking](), [latency](), [spark](), [speed](), [tracing]() ### AI overview An Adyen engineer investigates Cilium CNI timeouts that prevented new Kubernetes pods from starting. The article attributes the issue to sequential connection-tracking-table garbage collection, which became a bottleneck under workloads involving HDFS, Trino, and short-lived Spark jobs. ### Source excerpt By Jorrick Sleijster - Senior Data Platform Engineer, Adyen I was fully aware a year ago that a single configuration line could break the Kubernetes networking stack. But if they told me that leftovers from Kubernetes pods which terminated hours prior could block new ones from starting, I would have thought they were joking. In high-performance networking, 35 seconds is a lifetime. This was the latency required to iterate through our connection tracking table of 7 million entries at a maximum speed of 200,000 entries per second. At our 16-million-entry peak, this sequential lookup could take up to 80 seconds, leading to Cilium CNI timeouts preventing new pods from starting on affected nodes. We uncovered this linear-time behavior at Adyen by tracing syscalls, inspecting codebases, and analyzing eBPF internals. This investigation revealed how our varied workloads turned the connection tracking table's garbage collection algorithm into a critical bottleneck. Our setup: why we're different At Adyen, we run Cilium CNI across all our 100+ Kubernetes clusters. When we switched from Calico to Cilium, we knew we'd face challenges adapting it to our production workloads. Our production big data Kubernetes clusters have a unique usage pattern compared to the other Kubernetes environments within Adyen: Data extraction from HDFS. Our infrastructure relies on more than 500 datanodes. Trino represents one of our most demanding HDFS workloads, processing analytical queries against data stored on HDFS. Due to the distributed nature of HDFS, each file you download requires a new connection to any of these 500 nodes. Therefore, during peak hours, a single pod can produce approximately 50,000 connections every minute. Pod churn. Many pods we spawn on the Kubernetes cluster run batch jobs, such as Spark jobs. They stay around for anywhere from a second to a couple of hours. Wide variety of workloads. Some workloads are very CPU-intensive, like Spark pods executing complex joins and tra ## Implementing Data Validation with Great Expectations in Hybrid Environments DevFeed: [Implementing Data Validation with Great Expectations in Hybrid Environments]() Original publisher: [Read original article]() Author: Kamila Widyanto Full time DevOps; Site Reliability Engineer; Part Time Rubberduck Linkedin Profile Published: 2023-04-25T00:00:00Z Content type: tutorial Language: en Sources: [Trivago]() Topics: [data-processing](), [Hadoop](), [hdfs](), [integrity](), [Apache Spark](), [configuration](), [Library](), [Python](), [JSON](), [Open Source](), [YAML](), [version-control]() Tags: [configuration](), [data](), [data-pipeline](), [data-processing](), [data-validation](), [devops](), [distributed](), [hadoop](), [hdfs](), [integrity](), [json](), [library](), [open-source](), [python](), [version-control](), [workflow](), [yaml]() ### AI overview This article describes implementing Great Expectations for data validation in a hybrid Hadoop environment. It explains the framework's core concepts and how the authors ran it as a PySpark job in an automated data pipeline, including configuring the Data Context for HDFS constraints. ### Source excerpt Data validation is an essential step in any data processing pipeline, as it ensures the integrity and accuracy of the data to be used across all subsequent processing steps. ## 分布式文件系统的演化 DevFeed: [分布式文件系统的演化]() Original publisher: [Read original article]() Author: Joway Published: 2020-06-14T00:00:00Z Content type: tutorial Language: zh Sources: [Random Thoughts]() Topics: [IO](), [Linux](), [Ext4](), [hdfs](), [Google]() Tags: [blocks](), [client](), [ext4](), [filesystem](), [google](), [hdfs](), [io](), [lease](), [linux](), [master](), [pointers](), [primary](), [ssd](), [tech]() ### AI overview This article explains local filesystem structure, including superblocks, inodes, block pointers, and file-size limits, then introduces the evolution and architecture of distributed filesystems. It uses GFS to describe metadata lookup, chunk locations, caching, replication, leases, and client read and write flows, and mentions HDFS as a related implementation. ### Source excerpt 文件系统是操作系统 IO 栈里非常重要的一个中间层，其存在的意义是为了让上层应用程序有一层更加符合人类直觉的抽象来进行文档的读写，而无需考虑底层存储上的细节。 本地文件系统 在讨论分布式文件系统前，我们先来回顾下本地文件系统的组成。 存储结构 在前面一张图里，我们能够看到文件系统直接和通用块层进行交互，无论底层存储介质是磁盘还是 SSD，都被该层抽象为 Block 的概念。文件系统在初始化时，会先在挂载的块存储上的第一个位置创建一个 Super Block： ## ApacheCon Budapest 2014 DevFeed: [ApacheCon Budapest 2014]() Original publisher: [Read original article]() Author: mick Published: 2014-11-25T13:00:00Z Content type: opinion Language: en Sources: [Finn.no]() Topics: [Hadoop](), [Apache Spark](), [Docker](), [hdfs](), [Apache Cassandra](), [Streaming](), [couchdb](), [Data structures]() Tags: [apache](), [cassandra](), [couchdb](), [data-structures](), [docker](), [hadoop](), [hdfs](), [spark](), [streaming]() ### AI overview A firsthand account of ApacheCon Budapest 2014 discusses Apache communities and several technologies, including Docker, Spark, Cassandra, CouchDB, Hadoop, Mesos, Aurora, YARN, Solr, and OpenOffice. It highlights considerations around scalability, testing, streaming, production utilisation, and data processing. ### Source excerpt Last week was Europe's ApacheCon, held in Budapest. A refreshingly "laid-back and no-nonsense" conference free from the douchebaggery that comes from big sponsors and marketing. This year it was held in the awesome 5-star Corinthia Hotel, and consisted of a welcoming and cheerful vibe, a place for apache members, committers, and all, to get together and better know each other. Apache continues to grow as the foundation for strong communities building trusted open sourced solutions, for running half the internet, and providing a substantial amount of code to your java stack. Some hot topics were Docker, Mesos, Spark, Cassandra, CouchDB, Hadoop, Solr, OpenOffice, and the need for greater diversity in our communities. Spark is all the rage because of its brevity and simplicity, but isn't really a complete solution yet because it doesn't scale in many situations. Docker is awesome, and looks to take over the testing domain, but it remains limited and out of production as long as it has no network stack implementation. There was a very cool demonstration of Mesos and Aurora increasing production utilisation, even allowing in quiet periods for development/testing servers to come in. The next release of YARN (2.6) will see support for long-lived services, whereby we can transform many of our hadoop jobs with very little effort into true streaming solutions. YARN will also be able to deploy docker containers through its clusters, this could be a very nice solution for our batch jobs. Putting these together and you see that the technical separation between streaming and aggregating solutions fades away and it really just boils down to what data structures you solve each use case with. Talking with a hadoop committer it came to light that FINN could be running the fastest hadoop cluster that they know of, given HDFS runs on SSDs and is dedicated just for hadoop internals. Of no surprise Solr dealt a lot with scaling and performance, while Cassandra presented use-cases from ac